{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用seq2seq网络和注意力机制来实现机器翻译\n",
    "\n",
    "本示例会介绍使用seq2seq网络来实现机器翻译，同时使用注意力机制来提高seq2seq的效果(尤其是长句)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sequence to sequence模型\n",
    "\n",
    "sequence to sequence模型，或者说seq2seq模型，由两个RNN组成。这两个RNN分别叫做encoder和decoder。\n",
    "encoder会一个词一个出的读入输入序列，每一步都有一个输出，而最后的输出叫做context向量，我们可以认为是模型对源语言句子“语义”的一种表示。而decoder用这个context向量一步一步的生成目标语言的句子。\n",
    "\n",
    "![](seq2seq.png)\n",
    "\n",
    "为什么要两个RNN呢，如果我们使用一个RNN，输入和输出是一对一的关系（对于分类，我们可以只使用最后一个输出），但是翻译肯定不是一个词对一个词的翻译。当然这只是使用两个RNN在形式上的方便，从“原理”上来说，人类翻译也是类似的，首先仔细阅读源语句，然后“理解”它，而所谓的“理解”在seq2seq模型里可以认为encoding的过程，然后再根据理解，翻译成目标语句。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 注意力机制\n",
    "\n",
    "用一个固定长度的向量来承载输入序列的完整“语义”，不管向量能有多长，都是很困难的事情。\n",
    "\n",
    "[Bahdanau et al.等人引入的](https://arxiv.org/abs/1409.0473) **注意力机制**试图这样来解决这个问题：我们不依赖于一个固定长度的向量，而是通过“注意”输入的某些部分。在decoer每一步解码的时候都通过这个机制来选择输入的一部分来重点考虑。这似乎是合乎人类的翻译过程的——我们首先通过一个encoder大致理解句子的意思（编码到一个定长向量），具体翻译某个词或者短语的时候我们会仔细推敲对应的源语言的词（注意力机制）。\n",
    "\n",
    "![](5y6SCvU.png)\n",
    "\n",
    "注意力是通过decoder的另外一个神经网络层来计算的。它的输入是当前输入和上一个时刻的隐状态，输出是一个新的向量，这个向量的长度和输入相同（因为输入是变长的，我们会选择一个最大的长度），这个向量会用softmax变成“概率”，得到*注意力权重*，这个权重可以认为需要花费多大的“注意力”到某个输入上，因此我们会用这个权重加权平均encoder的输出，从而得到一个新的context向量，这个向量会用来预测当前时刻的输出。\n",
    "\n",
    "![](K1qMPxs.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 依赖"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unicodedata\n",
    "import string\n",
    "import re\n",
    "import random\n",
    "import time\n",
    "import math\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "from torch import optim\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#如果没有GPU请改成False\n",
    "USE_CUDA = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 加载数据\n",
    "\n",
    "训练数据是几千个英语到法语的平行句对。我们这里只是介绍算法，所以使用一个很小的数据集来演示。数据在data/eng-fra.txt里，格式如下：\n",
    "```\n",
    "I am cold.    Je suis froid.\n",
    "```\n",
    "我们会用one-hot的方法来表示一个单词。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 单词变成数字\n",
    "我们会创建一个Lang对象来表示源/目标语言，它包含word2idx、idx2word和word2count，分别表示单词到id、id到单词和单词的词频。\n",
    "word2count的作用是用于过滤一些低频词（把它变成unknown）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "SOS_token = 0\n",
    "EOS_token = 1\n",
    "\n",
    "class Lang:\n",
    "    def __init__(self, name):\n",
    "        self.name = name\n",
    "        self.word2index = {}\n",
    "        self.word2count = {}\n",
    "        self.index2word = {0: \"SOS\", 1: \"EOS\"}\n",
    "        self.n_words = 2\n",
    "      \n",
    "    def index_words(self, sentence):\n",
    "        for word in sentence.split(' '):\n",
    "            self.index_word(word)\n",
    "\n",
    "    def index_word(self, word):\n",
    "        if word not in self.word2index:\n",
    "            self.word2index[word] = self.n_words\n",
    "            self.word2count[word] = 1\n",
    "            self.index2word[self.n_words] = word\n",
    "            self.n_words += 1\n",
    "        else:\n",
    "            self.word2count[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unicode_to_ascii(s):\n",
    "    return ''.join(\n",
    "        c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn'\n",
    "    )\n",
    "\n",
    "# 大小转小写，trim，去掉非字母。\n",
    "def normalize_string(s):\n",
    "    s = unicode_to_ascii(s.lower().strip())\n",
    "    s = re.sub(r\"([.!?])\", r\" \\1\", s)\n",
    "    s = re.sub(r\"[^a-zA-Z.!?]+\", r\" \", s)\n",
    "    return s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 读取文件\n",
    "\n",
    "文件格式是每行一个句对，用tab分隔，第一个是英语，第二个是法语，为了方便未来的复用，我们有一个reverse参数，这样如果我们需要法语到英语的翻译就可以用到。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_langs(lang1, lang2, reverse=False):\n",
    "    print(\"Reading lines...\")\n",
    "\n",
    "    # 读取文件\n",
    "    lines = open('../data/%s-%s.txt' % (lang1, lang2)).read().strip().split('\\n')\n",
    "    \n",
    "    # split\n",
    "    pairs = [[normalize_string(s) for s in l.split('\\t')] for l in lines]\n",
    "    \n",
    "    if reverse:\n",
    "        pairs = [list(reversed(p)) for p in pairs]\n",
    "        input_lang = Lang(lang2)\n",
    "        output_lang = Lang(lang1)\n",
    "    else:\n",
    "        input_lang = Lang(lang1)\n",
    "        output_lang = Lang(lang2)\n",
    "        \n",
    "    return input_lang, output_lang, pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 过滤句子\n",
    "\n",
    "作为演示，我们只挑选长度小于10的句子，并且这保留\"I am\"和\"He is\"开头的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LENGTH = 10\n",
    "\n",
    "good_prefixes = (\n",
    "    \"i am \", \"i m \",\n",
    "    \"he is\", \"he s \",\n",
    "    \"she is\", \"she s\",\n",
    "    \"you are\", \"you re \"\n",
    ")\n",
    "\n",
    "def filter_pair(p):\n",
    "    return len(p[0].split(' ')) < MAX_LENGTH and len(p[1].split(' ')) < MAX_LENGTH and \\\n",
    "        p[1].startswith(good_prefixes)\n",
    "\n",
    "def filter_pairs(pairs):\n",
    "    return [pair for pair in pairs if filter_pair(pair)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据处理过程如下：\n",
    "\n",
    "* 读取文件，split成行，再split成pair\n",
    "* 文本归一化，通过长度和内容过滤\n",
    "* 通过pair里的句子得到单词列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading lines...\n",
      "Read 149861 sentence pairs\n",
      "Trimmed to 9738 sentence pairs\n",
      "Indexing words...\n",
      "['il a la grippe .', 'he s got the flu .']\n"
     ]
    }
   ],
   "source": [
    "def prepare_data(lang1_name, lang2_name, reverse=False):\n",
    "    input_lang, output_lang, pairs = read_langs(lang1_name, lang2_name, reverse)\n",
    "    print(\"Read %s sentence pairs\" % len(pairs))\n",
    "    \n",
    "    pairs = filter_pairs(pairs)\n",
    "    print(\"Trimmed to %s sentence pairs\" % len(pairs))\n",
    "    \n",
    "    print(\"Indexing words...\")\n",
    "    for pair in pairs:\n",
    "        input_lang.index_words(pair[0])\n",
    "        output_lang.index_words(pair[1])\n",
    "\n",
    "    return input_lang, output_lang, pairs\n",
    "\n",
    "input_lang, output_lang, pairs = prepare_data('eng', 'fra', True)\n",
    " \n",
    "print(random.choice(pairs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 把训练数据变成Tensor/Variable\n",
    "\n",
    "为了让神经网络能够处理，我们首先需要把句子变成Tensor。每个句子首先分成词，每个词被替换成对应的index。另外我们会增加一个特殊的EOS来表示句子的结束。\n",
    "\n",
    "![](https://i.imgur.com/LzocpGH.png)\n",
    "\n",
    "在PyTorch里一个Tensor是一个多维数组，它的所有元素的数据类型都是一样的。我们这里使用LongTensor来表示词的index。\n",
    "\n",
    "可以训练的PyTorch模块要求输入是Variable而不是Tensor。变量除了包含Tensor的内容之外，它还会跟踪计算图的状态，从而可以进行自动梯度的求值。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def indexes_from_sentence(lang, sentence):\n",
    "    return [lang.word2index[word] for word in sentence.split(' ')]\n",
    "\n",
    "def variable_from_sentence(lang, sentence):\n",
    "    indexes = indexes_from_sentence(lang, sentence)\n",
    "    indexes.append(EOS_token)\n",
    "    var = Variable(torch.LongTensor(indexes).view(-1, 1))\n",
    "\n",
    "    if USE_CUDA: var = var.cuda()\n",
    "    return var\n",
    "\n",
    "def variables_from_pair(pair):\n",
    "    input_variable = variable_from_sentence(input_lang, pair[0])\n",
    "    target_variable = variable_from_sentence(output_lang, pair[1])\n",
    "    return (input_variable, target_variable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./encoder-network.png\" style=\"float: right\" />\n",
    "\n",
    "seq2seq网络的encoder是一个RNN，对于输入的每一个词都输出一个向量和一个隐状态，这个隐状态会作为下一个时刻的输入。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderRNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, n_layers=1):\n",
    "        super(EncoderRNN, self).__init__()\n",
    "        \n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.n_layers = n_layers\n",
    "        \n",
    "        self.embedding = nn.Embedding(input_size, hidden_size)\n",
    "        self.gru = nn.GRU(hidden_size, hidden_size, n_layers)\n",
    "        \n",
    "    def forward(self, word_inputs, hidden):\n",
    "        # 注意：和名字分类不同，我们这里一次处理完一个输入的所有词，而不是for循环每次处理一个词。\n",
    "        # 两者的效果是一样的，但是一次处理万效率更高。\n",
    "        seq_len = len(word_inputs)\n",
    "        embedded = self.embedding(word_inputs).view(seq_len, 1, -1)\n",
    "        output, hidden = self.gru(embedded, hidden)\n",
    "        return output, hidden\n",
    "\n",
    "    def init_hidden(self):\n",
    "        hidden = Variable(torch.zeros(self.n_layers, 1, self.hidden_size))\n",
    "        if USE_CUDA: hidden = hidden.cuda()\n",
    "        return hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 理解Bahdanau等人提出的模型\n",
    "\n",
    "下面我们来学习一下这篇文章提出的 [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473) Attention Decoder。\n",
    "\n",
    "decoder在每一个时刻的输出依赖与前一个时刻的输出和一个$\\mathbf x$，这个$\\mathbf x$包括当前的隐状态（它也会考虑前一个时刻的输出）和一个注意力”context“，下文会介绍它。函数$g$是一个带非线性激活的全连接层，它的输入是$y_{i-1}$, $s_i$ 和 $c_i$拼接起来的。\n",
    "\n",
    "$$\n",
    "p(y_i \\mid \\{y_1,...,y_{i-1}\\},\\mathbf{x}) = g(y_{i-1}, s_i, c_i)\n",
    "$$\n",
    "\n",
    "上式的意思是：我们在翻译当前词时只考虑上一个翻译出来的词以及当前的隐状态和注意力context。\n",
    "\n",
    "当前隐状态$s_i$是有RNN $f$计算出来的，这个RNN的输入是上一个隐状态$s_{i-1}$，decoder的上一个输出$y_{i-1}$和 context向量$c_i$。\n",
    "\n",
    "在代码实现中，我们使用的RNN是`nn.GRU`，隐状态 $s_i$是`hidden`，输出$y_i$是`output`， context $c_i$ 是`context`。\n",
    "\n",
    "$$\n",
    "s_i = f(s_{i-1}, y_{i-1}, c_i)\n",
    "$$\n",
    "\n",
    "context向量$c_i$是encoder在每个时刻(词)的输出的加权和，而权值$a_{ij}$表示i时刻需要关注$h_j$的程度(概率)。\n",
    "\n",
    "$$\n",
    "c_i = \\sum_{j=1}^{T_x} a_{ij} h_j\n",
    "$$\n",
    "\n",
    "而权值$a_{ij}$是\"能量\" $e_{ij}$的softmax。\n",
    "\n",
    "$$\n",
    "a_{ij} = \\dfrac{exp(e_{ij})}{\\sum_{k=1}^{T} exp(e_{ik})}\n",
    "$$\n",
    "\n",
    "而能量$e_{ij}$是上个时刻的隐状态$s_{i-1}$和encoder第j个时刻的输出$h_j$的函数：\n",
    "\n",
    "$$\n",
    "e_{ij} = a(s_{i-1}, h_j)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实现Bahdanau模型\n",
    "\n",
    "总结一下，我们的decoder有4个主要的部分——一个embedding层用于把输入词变成向量；一个用于计算注意力能量的层；一个RNN层；一个输出层。\n",
    "\n",
    "decoder的输入是上一个时刻的隐状态$s_{i-1}$，上一个时刻的输出$y_{i-1}$和所有encoder的输出$h_*$。\n",
    "\n",
    "* embedding层其输入是上一个时刻的输出$y_{i-1}$\n",
    "    * `embedded = embedding(last_rnn_output)`\n",
    "* attention层首先根据函数$a$计算e，其输入是$(s_{i-1}, h_j)$输出是$e_{ij}$，最后对e用softmax得到$a_{ij}$\n",
    "    * `attn_energies[j] = attn_layer(last_hidden, encoder_outputs[j])`\n",
    "    * `attn_weights = normalize(attn_energies)`\n",
    "* context向量$c_i$是encoder的所有时刻的输出的加权和\n",
    "    * `context = sum(attn_weights * encoder_outputs)`\n",
    "* RNN层$f$的输入是$(s_{i-1}, y_{i-1}, c_i)$和内部的隐状态，输出是$s_i$\n",
    "    * `rnn_input = concat(embedded, context)`\n",
    "    * `rnn_output, rnn_hidden = rnn(rnn_input, last_hidden)`\n",
    "* 输出层$g$的输入是$(y_{i-1}, s_i, c_i)$，输出是$y_i$\n",
    "    * `output = out(embedded, rnn_output, context)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 注意：我们后面并没有用到这个模型，代码只是为了让读者更加理解前面的内容。\n",
    "class BahdanauAttnDecoderRNN(nn.Module):\n",
    "    def __init__(self, hidden_size, output_size, n_layers=1, dropout_p=0.1):\n",
    "        super(AttnDecoderRNN, self).__init__()\n",
    "        \n",
    "        # 定义参数\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.dropout_p = dropout_p\n",
    "        self.max_length = max_length\n",
    "        \n",
    "        # 定义网络\n",
    "        self.embedding = nn.Embedding(output_size, hidden_size)\n",
    "        self.dropout = nn.Dropout(dropout_p)\n",
    "        self.attn = GeneralAttn(hidden_size)\n",
    "        self.gru = nn.GRU(hidden_size * 2, hidden_size, n_layers, dropout=dropout_p)\n",
    "        self.out = nn.Linear(hidden_size, output_size)\n",
    "    \n",
    "    def forward(self, word_input, last_hidden, encoder_outputs):\n",
    "        # 每次decode我们只运行一个time step，但是会使用encoder的所有输出\n",
    "        \n",
    "        # 得到当前词(decoder的上一个输出)的embedding\n",
    "        word_embedded = self.embedding(word_input).view(1, 1, -1) # S=1 x B x N\n",
    "        word_embedded = self.dropout(word_embedded)\n",
    "        \n",
    "        # 计算attention weights\n",
    "        attn_weights = self.attn(last_hidden[-1], encoder_outputs)\n",
    "        context = attn_weights.bmm(encoder_outputs.transpose(0, 1)) # B x 1 x N\n",
    "        \n",
    "        # 把word_embedded和context拼接起来作为rnn的输入\n",
    "        rnn_input = torch.cat((word_embedded, context), 2)\n",
    "        output, hidden = self.gru(rnn_input, last_hidden)\n",
    "        \n",
    "        # 输出层\n",
    "        output = output.squeeze(0) # B x N\n",
    "        output = F.log_softmax(self.out(torch.cat((output, context), 1)), 1)\n",
    "        \n",
    "        # 返回结果\n",
    "        return output, hidden, attn_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Luong等人提出的模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luong等人在[Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/abs/1508.04025)提出了更多的提高和简化。他们描述了”全局注意力“模型，其计算注意力得分的方法和之前不同。前面是通过$s_{i-1}$和$h_j$计算$a_{ij}$，也就是当前的注意力权重依赖与前一个状态，而这里的注意力依赖与decoder当前的隐状态和encoder所有隐状态：\n",
    "\n",
    "$$\n",
    "a_t(s) = align(h_t, \\bar h_s)  = \\dfrac{exp(score(h_t, \\bar h_s))}{\\sum_{s'} exp(score(h_t, \\bar h_{s'}))}\n",
    "$$\n",
    "\n",
    "特点的\"score\"函数会比较两个隐状态的”相似度“，可以是两个向量的内积，也可以是$h_{s'}$做一个线性变换之后和$h_t$的内积，也可以是把两个向量拼接起来然后做一个线性变换，然后和一个参数$v_a$（这个参数是学习出来的）的内积：\n",
    "\n",
    "$$\n",
    "score(h_t, \\bar h_s) =\n",
    "\\begin{cases}\n",
    "h_t ^\\top \\bar h_s & dot \\\\\n",
    "h_t ^\\top \\textbf{W}_a \\bar h_s & general \\\\\n",
    "v_a ^\\top \\textbf{W}_a [ h_t ; \\bar h_s ] & concat\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "scoring函数的模块化定义使得我们可以随意的修改而不影响其它地方的代码。这个模块的输入总是decoder的隐状态和encoder的所有输出。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Attn(nn.Module):\n",
    "    def __init__(self, method, hidden_size, max_length=MAX_LENGTH):\n",
    "        super(Attn, self).__init__()\n",
    "        \n",
    "        self.method = method\n",
    "        self.hidden_size = hidden_size\n",
    "        \n",
    "        if self.method == 'general':\n",
    "            self.attn = nn.Linear(self.hidden_size, hidden_size)\n",
    "\n",
    "        elif self.method == 'concat':\n",
    "            self.attn = nn.Linear(self.hidden_size * 2, hidden_size)\n",
    "            self.other = nn.Parameter(torch.FloatTensor(1, hidden_size))\n",
    "\n",
    "    def forward(self, hidden, encoder_outputs):\n",
    "        seq_len = len(encoder_outputs)\n",
    "\n",
    "        # 创建变量来存储注意力能量\n",
    "        attn_energies = Variable(torch.zeros(seq_len))\n",
    "        if USE_CUDA: attn_energies = attn_energies.cuda()\n",
    "\n",
    "        # 计算\n",
    "        for i in range(seq_len):\n",
    "            attn_energies[i] = self.score(hidden, encoder_outputs[i])\n",
    "\n",
    "        return F.softmax(attn_energies, 0).unsqueeze(0).unsqueeze(0)\n",
    "    \n",
    "    def score(self, hidden, encoder_output):\n",
    "\n",
    "        if self.method == 'dot':\n",
    "            energy = hidden.view(-1).dot(encoder_output.view(-1))\n",
    "            return energy\n",
    "\n",
    "        elif self.method == 'general':\n",
    "            energy = self.attn(encoder_output)\n",
    "            energy = hidden.view(-1).dot(energy.view(-1))\n",
    "            return energy\n",
    "\n",
    "        elif self.method == 'concat':\n",
    "            energy = self.attn(torch.cat((hidden, encoder_output), 1))\n",
    "            energy = self.other.view(-1).dot(energy.view(-1))\n",
    "            return energy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们可以构建一个decoder，它会把Attn模块放到RNN之后用来计算注意力的权重，并且用它来计算context向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttnDecoderRNN(nn.Module):\n",
    "    def __init__(self, attn_model, hidden_size, output_size, n_layers=1, dropout_p=0.1):\n",
    "        super(AttnDecoderRNN, self).__init__()\n",
    "        \n",
    "        # 保存到self里\n",
    "        self.attn_model = attn_model\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.dropout_p = dropout_p\n",
    "        \n",
    "        # 定义网络中的层\n",
    "        self.embedding = nn.Embedding(output_size, hidden_size)\n",
    "        self.gru = nn.GRU(hidden_size * 2, hidden_size, n_layers, dropout=dropout_p)\n",
    "        self.out = nn.Linear(hidden_size * 2, output_size)\n",
    "        \n",
    "        # 选择注意力模型\n",
    "        if attn_model != 'none':\n",
    "            self.attn = Attn(attn_model, hidden_size)\n",
    "    \n",
    "    def forward(self, word_input, last_context, last_hidden, encoder_outputs):\n",
    "        # 注意：每次我们处理一个time step\n",
    "        \n",
    "        # 得到当前输入(上一个输出)的embedding\n",
    "        word_embedded = self.embedding(word_input).view(1, 1, -1) # S=1 x B x N\n",
    "        \n",
    "        # 把当前的embedding和上一个context拼接起来输入到RNN里\n",
    "        rnn_input = torch.cat((word_embedded, last_context.unsqueeze(0)), 2)\n",
    "        rnn_output, hidden = self.gru(rnn_input, last_hidden)\n",
    "\n",
    "        # 使用RNN的输出和所有encoder的输出来计算注意力权重，然后计算context向量\n",
    "        attn_weights = self.attn(rnn_output.squeeze(0), encoder_outputs)\n",
    "        context = attn_weights.bmm(encoder_outputs.transpose(0, 1)) # B x 1 x N\n",
    "        \n",
    "        # 使用RNN的输出和context向量预测下一个词\n",
    "        rnn_output = rnn_output.squeeze(0) # S=1 x B x N -> B x N\n",
    "        context = context.squeeze(1)       # B x S=1 x N -> B x N\n",
    "        output = F.log_softmax(self.out(torch.cat((rnn_output, context), 1)), 1)\n",
    "        \n",
    "        return output, context, hidden, attn_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 测试模型 \n",
    "\n",
    "为了验证代码是否有问题，我们可以用fake的数据来测试一下它的输出："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EncoderRNN(\n",
      "  (embedding): Embedding(10, 10)\n",
      "  (gru): GRU(10, 10, num_layers=2)\n",
      ")\n",
      "AttnDecoderRNN(\n",
      "  (embedding): Embedding(10, 10)\n",
      "  (gru): GRU(20, 10, num_layers=2, dropout=0.1)\n",
      "  (out): Linear(in_features=20, out_features=10, bias=True)\n",
      "  (attn): Attn(\n",
      "    (attn): Linear(in_features=10, out_features=10, bias=True)\n",
      "  )\n",
      ")\n",
      "torch.Size([1, 10]) torch.Size([2, 1, 10]) torch.Size([1, 1, 3])\n",
      "torch.Size([1, 10]) torch.Size([2, 1, 10]) torch.Size([1, 1, 3])\n",
      "torch.Size([1, 10]) torch.Size([2, 1, 10]) torch.Size([1, 1, 3])\n"
     ]
    }
   ],
   "source": [
    "#USE_CUDA = True #如果没有GPU，改成False\n",
    "encoder_test = EncoderRNN(10, 10, 2)\n",
    "decoder_test = AttnDecoderRNN('general', 10, 10, 2)\n",
    "print(encoder_test)\n",
    "print(decoder_test)\n",
    "\n",
    "encoder_hidden = encoder_test.init_hidden()\n",
    "word_input = Variable(torch.LongTensor([1, 2, 3]))\n",
    "if USE_CUDA:\n",
    "    encoder_test.cuda()\n",
    "    word_input = word_input.cuda()\n",
    "encoder_outputs, encoder_hidden = encoder_test(word_input, encoder_hidden)\n",
    "\n",
    "word_inputs = Variable(torch.LongTensor([1, 2, 3]))\n",
    "decoder_attns = torch.zeros(1, 3, 3)\n",
    "decoder_hidden = encoder_hidden\n",
    "decoder_context = Variable(torch.zeros(1, decoder_test.hidden_size))\n",
    "\n",
    "if USE_CUDA:\n",
    "    decoder_test.cuda()\n",
    "    word_inputs = word_inputs.cuda()\n",
    "    decoder_context = decoder_context.cuda()\n",
    "\n",
    "for i in range(3):\n",
    "    decoder_output, decoder_context, decoder_hidden, decoder_attn = decoder_test(word_inputs[i], decoder_context, decoder_hidden, encoder_outputs)\n",
    "    print(decoder_output.size(), decoder_hidden.size(), decoder_attn.size())\n",
    "    decoder_attns[0, i] = decoder_attn.squeeze(0).cpu().data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练\n",
    "\n",
    "### 一次训练\n",
    "\n",
    "对于一个训练数据，我们首先用encoder对输入句子进行编码，得到每个时刻的输出和最后一个时刻的隐状态。最后一个隐状态会作为decoder隐状态的初始值，并且我们会用一个特殊的`<SOS>`作为decoder的第一个输入。\n",
    "\n",
    "### Teacher Forcing 和 Scheduled Sampling\n",
    "\n",
    "\"Teacher Forcing\"，或者叫最大似然采样，使用目标语言的实际输出来作为decoder的输入。而另外一种方法就是使用decoder上一个时刻的输出来作为当前时刻的输入。前者可以让网络更快收敛，但是根据这篇论文(http://minds.jacobs-university.de/sites/default/files/uploads/papers/ESNTutorialRev.pdf)，它可能不稳定。\n",
    "\n",
    "实践中发现，Teacher Forcing可以学到合法语法的翻译，但是效果很不好。\n",
    "\n",
    "解决方法是[Scheduled Sampling](https://arxiv.org/abs/1506.03099)，随机的使用目标语言的输出和decoder预测的输出。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "teacher_forcing_ratio = 0.5\n",
    "clip = 5.0\n",
    "\n",
    "def train(input_variable, target_variable, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion, max_length=MAX_LENGTH):\n",
    "\n",
    "    # 梯度清零\n",
    "    encoder_optimizer.zero_grad()\n",
    "    decoder_optimizer.zero_grad()\n",
    "    loss = 0 \n",
    "\n",
    "    # 得到输入和输出句子的长度\n",
    "    input_length = input_variable.size()[0]\n",
    "    target_length = target_variable.size()[0]\n",
    "\n",
    "    # encoding\n",
    "    encoder_hidden = encoder.init_hidden()\n",
    "    encoder_outputs, encoder_hidden = encoder(input_variable, encoder_hidden)\n",
    "    \n",
    "    # 准备输入和输出变量\n",
    "    decoder_input = Variable(torch.LongTensor([[SOS_token]]))\n",
    "    decoder_context = Variable(torch.zeros(1, decoder.hidden_size))\n",
    "    decoder_hidden = encoder_hidden # Use last hidden state from encoder to start decoder\n",
    "    if USE_CUDA:\n",
    "        decoder_input = decoder_input.cuda()\n",
    "        decoder_context = decoder_context.cuda()\n",
    "\n",
    "    # 随机选择是否Teacher Forcing\n",
    "    use_teacher_forcing = random.random() < teacher_forcing_ratio\n",
    "    if use_teacher_forcing:\n",
    "        \n",
    "        # Teacher forcing：使用真实的输出作为下一个时刻的输入\n",
    "        for di in range(target_length):\n",
    "            decoder_output, decoder_context, decoder_hidden, decoder_attention = decoder(decoder_input, decoder_context, decoder_hidden, encoder_outputs)\n",
    "            loss += criterion(decoder_output, target_variable[di])\n",
    "            decoder_input = target_variable[di] # 下一个时刻的输入来自target\n",
    "\n",
    "    else:\n",
    "        # 不使用 teacher forcing：使用decoder的预测作为下一个时刻的输入\n",
    "        for di in range(target_length):\n",
    "            decoder_output, decoder_context, decoder_hidden, decoder_attention = decoder(decoder_input, decoder_context, decoder_hidden, encoder_outputs)\n",
    "            loss += criterion(decoder_output, target_variable[di])\n",
    "            \n",
    "            # 选择最可能的词\n",
    "            topv, topi = decoder_output.data.topk(1)\n",
    "            ni = topi.item()\n",
    "            \n",
    "            decoder_input = Variable(torch.LongTensor([[ni]])) # 下个时刻的输入\n",
    "            if USE_CUDA: decoder_input = decoder_input.cuda()\n",
    "\n",
    "            # 如果decoder输出EOS_token，那么提前结束\n",
    "            if ni == EOS_token: break\n",
    "\n",
    "    # 反向计算梯度\n",
    "    loss.backward()\n",
    "    torch.nn.utils.clip_grad_norm_(encoder.parameters(), clip)\n",
    "    torch.nn.utils.clip_grad_norm_(decoder.parameters(), clip)\n",
    "    encoder_optimizer.step()\n",
    "    decoder_optimizer.step()\n",
    "    \n",
    "    return loss.item() / target_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "辅助函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def as_minutes(s):\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)\n",
    "\n",
    "def time_since(since, percent):\n",
    "    now = time.time()\n",
    "    s = now - since\n",
    "    es = s / (percent)\n",
    "    rs = es - s\n",
    "    return '%s (- %s)' % (as_minutes(s), as_minutes(rs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 进行训练\n",
    "\n",
    "所有准备工作都好了，我们可以初始化网络并开始训练了。\n",
    "\n",
    "我们首先初始化网络，优化器和损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "attn_model = 'general'\n",
    "hidden_size = 500\n",
    "n_layers = 2\n",
    "dropout_p = 0.05\n",
    "\n",
    "# 初始化模型\n",
    "encoder = EncoderRNN(input_lang.n_words, hidden_size, n_layers)\n",
    "decoder = AttnDecoderRNN(attn_model, hidden_size, output_lang.n_words, n_layers, dropout_p=dropout_p)\n",
    "\n",
    "# 把变量放到GPU里\n",
    "if USE_CUDA:\n",
    "    encoder.cuda()\n",
    "    decoder.cuda()\n",
    "\n",
    "# 初始化optimizer和criterion，注意：我们这里用两个optimizer分别来优化encoder和decoder的参数。\n",
    "learning_rate = 0.0001\n",
    "encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate)\n",
    "decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate)\n",
    "criterion = nn.NLLLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义一些变量用于保存历史的loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 训练的一些超参数\n",
    "n_epochs = 50000\n",
    "plot_every = 200\n",
    "print_every = 1000\n",
    "\n",
    "# 保存开始时间和loss\n",
    "start = time.time()\n",
    "plot_losses = []\n",
    "print_loss_total = 0 \n",
    "plot_loss_total = 0 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0m 47s (- 38m 42s) (1000 2%) 3.2142\n",
      "1m 38s (- 39m 20s) (2000 4%) 2.8133\n",
      "2m 29s (- 39m 8s) (3000 6%) 2.6346\n",
      "3m 20s (- 38m 26s) (4000 8%) 2.5035\n",
      "4m 12s (- 37m 52s) (5000 10%) 2.4569\n",
      "5m 3s (- 37m 5s) (6000 12%) 2.3371\n",
      "5m 55s (- 36m 22s) (7000 14%) 2.2214\n",
      "6m 47s (- 35m 41s) (8000 16%) 2.0350\n",
      "7m 39s (- 34m 53s) (9000 18%) 2.0067\n",
      "8m 31s (- 34m 5s) (10000 20%) 1.9128\n",
      "9m 22s (- 33m 14s) (11000 22%) 1.8261\n",
      "10m 13s (- 32m 23s) (12000 24%) 1.7364\n",
      "11m 4s (- 31m 29s) (13000 26%) 1.7320\n",
      "11m 55s (- 30m 40s) (14000 28%) 1.6480\n",
      "12m 44s (- 29m 44s) (15000 30%) 1.5951\n",
      "13m 34s (- 28m 50s) (16000 32%) 1.5419\n",
      "14m 23s (- 27m 57s) (17000 34%) 1.4807\n",
      "15m 16s (- 27m 9s) (18000 36%) 1.4608\n",
      "16m 11s (- 26m 24s) (19000 38%) 1.3515\n",
      "17m 2s (- 25m 34s) (20000 40%) 1.3469\n",
      "17m 54s (- 24m 44s) (21000 42%) 1.2232\n",
      "18m 46s (- 23m 53s) (22000 44%) 1.2776\n",
      "19m 38s (- 23m 3s) (23000 46%) 1.2623\n",
      "20m 41s (- 22m 25s) (24000 48%) 1.1689\n",
      "21m 34s (- 21m 34s) (25000 50%) 1.1875\n",
      "22m 27s (- 20m 43s) (26000 52%) 1.0911\n",
      "23m 20s (- 19m 52s) (27000 54%) 1.1138\n",
      "24m 11s (- 19m 0s) (28000 56%) 1.0712\n",
      "25m 2s (- 18m 8s) (29000 57%) 1.0272\n",
      "25m 55s (- 17m 16s) (30000 60%) 0.9903\n",
      "26m 47s (- 16m 25s) (31000 62%) 0.8805\n",
      "27m 40s (- 15m 34s) (32000 64%) 0.9175\n",
      "28m 32s (- 14m 42s) (33000 66%) 0.9425\n",
      "29m 23s (- 13m 49s) (34000 68%) 0.8564\n",
      "30m 13s (- 12m 57s) (35000 70%) 0.8459\n",
      "31m 3s (- 12m 4s) (36000 72%) 0.8755\n",
      "31m 54s (- 11m 12s) (37000 74%) 0.8545\n",
      "32m 44s (- 10m 20s) (38000 76%) 0.7991\n",
      "33m 33s (- 9m 28s) (39000 78%) 0.7610\n",
      "34m 25s (- 8m 36s) (40000 80%) 0.7796\n",
      "35m 18s (- 7m 44s) (41000 82%) 0.7193\n",
      "36m 10s (- 6m 53s) (42000 84%) 0.7297\n",
      "37m 4s (- 6m 2s) (43000 86%) 0.6566\n",
      "37m 57s (- 5m 10s) (44000 88%) 0.6820\n",
      "38m 50s (- 4m 18s) (45000 90%) 0.6584\n",
      "39m 41s (- 3m 27s) (46000 92%) 0.6479\n",
      "40m 34s (- 2m 35s) (47000 94%) 0.6570\n",
      "41m 25s (- 1m 43s) (48000 96%) 0.6284\n",
      "42m 15s (- 0m 51s) (49000 98%) 0.5816\n",
      "43m 6s (- 0m 0s) (50000 100%) 0.5723\n"
     ]
    }
   ],
   "source": [
    "\n",
    "for epoch in range(1, n_epochs + 1):\n",
    "    \n",
    "    # 得到一个训练句对\n",
    "    training_pair = variables_from_pair(random.choice(pairs))\n",
    "    input_variable = training_pair[0]\n",
    "    target_variable = training_pair[1]\n",
    "\n",
    "    # 训练一次\n",
    "    loss = train(input_variable, target_variable, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)\n",
    "\n",
    "    print_loss_total += loss\n",
    "    plot_loss_total += loss\n",
    "\n",
    "    if epoch == 0: continue\n",
    "\n",
    "    if epoch % print_every == 0:\n",
    "        print_loss_avg = print_loss_total / print_every\n",
    "        print_loss_total = 0\n",
    "        print_summary = '%s (%d %d%%) %.4f' % (time_since(start, epoch / n_epochs), epoch, epoch / n_epochs * 100, print_loss_avg)\n",
    "        print(print_summary)\n",
    "\n",
    "    if epoch % plot_every == 0:\n",
    "        plot_loss_avg = plot_loss_total / plot_every\n",
    "        plot_losses.append(plot_loss_avg)\n",
    "        plot_loss_total = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 绘图训练loss\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd4XMXV+PHv2V313qwuy71XZGNTDRgwDjVAAiEJyUvCj4TwkpCQhLQ3PSGEdCChhZIAoYVeDAZj44p7L7Jlyeq9d2l+f9zdtbrW1pVkyefzPHrQ7p29O5eFs6O5c86IMQallFKji2O4O6CUUsp+GtyVUmoU0uCulFKjkAZ3pZQahTS4K6XUKKTBXSmlRiEN7kopNQppcFdKqVHI5+AuIk4R2SYib/TR5loRMSKSYU/3lFJKnQzXCbS9E9gHhPd0UETC3G02+nKy2NhYk56efgJvr5RSasuWLaXGmLj+2vkU3EUkBfgU8Cvgrl6a/QK4F7jbl3Omp6ezefNmX5oqpZRyE5FsX9r5Oi3zJ+C7QHsvbzYfSDXGvOnj+ZRSSg2ifoO7iFwOFBtjtvRy3AH8Afi2D+e6VUQ2i8jmkpKSE+6sUkop3/gycj8buFJEjgLPAReKyL86HA8DZgKr3G0WAa/1dFPVGPOwMSbDGJMRF9fvlJFSSqmT1G9wN8bcY4xJMcakAzcAHxhjPt/heJUxJtYYk+5uswG40hijE+pKKTVMTnqdu4j8XESutLMzSiml7HEiSyExxqwCVrl//0kvbZYMtFNKKaUGRjNUlVJqFLIlQ1VE7hKRvSKyU0RWishYe7t53IHCGu5fcYCy2qbBegullBrxTmTk7slQ7ck2IMMYMxt4EfjdQDvWm8ziWv76QSaltc2D9RZKKTXi+RTcO2SoPtrTcWPMh8aYevfDDUCKPd3rzuUUAFraesynUkophU0Zql3cArx90j3qh7/T6rIGd6WU6t2AM1S7tP08kAHc18vxAWeoekbure3mpF6vlFKnAzsyVAEQkaXAD7ESmHq822lHhqrLoSN3pZTqz4AzVAFEZB7wD6zAXjwoPXXz84zc23TkrpRSvbErQ/U+IBR4QUS2i8hrtvSuBy73nHtru47clVKqN7ZkqBpjltraqz64HJ7VMjpyV0qp3oy4DFV/l865K6VUf0ZccPeM3HXOXSmlemdX+YEAEfmPiGSKyEYRSbezkx356Tp3pZTql13lB24BKowxE4E/Yu2lOih0nbtSSvXPlvIDwFXAk+7fXwQuEhEZePe686xzb9WRu1JK9cqu8gPJwDEAY0wrUAXEDLh3PfCUH2jWOXellOqVreUHfDiXfeUHdOSulFK9sqv8QB6QCiAiLiACKOt6IlvKD+icu1JK9cuW8gPAa8DN7t+vc7cZlOjrp7VllFKqXyeUodqRiPwc2GyMeQ14DHhaRDKBcqwvgUHhcAgO0XXuSinVF7vKDzQC19vZsb64nA5atLaMUkr1asRlqIK1YqalVUfuSinVmxEZ3F1O0aqQSinVB1+WQgaKyCYR2SEie0TkZz20SRORD93lCXaKyPLB6a7F5XBoVUillOqDLyP3JuBCY8wcYC6wTEQWdWnzI+B5Y8w8rJupD9rbzc78nKLr3JVSqg/93lB1L2msdT/0c/90HTYbINz9ewSQb1cHe2JNy+jIXSmleuNrbRmniGwHioH3jDEbuzT5KfB5EckF3gLu6OU8A85QBasypK5zV0qp3vkU3I0xbcaYuUAKsFBEZnZpciPwhDEmBViOtea927ntyFAFK5FJg7tSSvXuhFbLGGMqgQ+BZV0O3QI8726zHggEYu3oYE9cTtEkJqWU6oMvq2XiRCTS/XsQcDGwv0uzHOAid5tpWMH95Odd+mElMWlwV0qp3viSoZoIPCkiTqwvg+eNMW90KT/wbeAREfkW1s3VLw1WbRkAP4eullFKqb74slpmJzCvh+c7lh/Yi1U9ckjotIxSSvVtRGao+jkdNOvIXSmlemVLhqq73WdEZK+7zTP2d/U4P6dDyw8opVQffJlz92So1oqIH/CxiLxtjNngaSAik4B7gLONMRUiMmaQ+guAy6HTMkop1Re7MlS/CjxgjKlwv6bYzk52pUlMSinVN7syVCcDk0VkrYhsEJGu6+BtpeUHlFKqb3ZlqLqAScASrGzVRzxr4zuyq/yAy+HQaRmllOqDXRmqucBrxpgWY0wWcBAr2Hd9vS3lB/xdoqtllFKqD3ZlqL6CNWpHRGKxpmmO2NrTDqyRuwZ3pZTqjV0Zqu8Cl4jIXqANuNsYUzZondYkJqWU6pNdGaoGuMv9M+j8dINspZTq04jMUNV17kop1bcRGdytDFXDINYmU0qpEc228gPutteKiBGRDHu72ZmfUwB0k2yllOqFLeUHAEQkDLgT6JrgZDuX0/pOam1vx39k/vGhlFKDqt/IaCz9lR8A+AVwL9BoX/d65nLoyF0ppfpiS/kBEZkPpBpj3uznPLZtkA3oWnellOrFgMsPuDfC/gPWbkz9nceWDFWXe85d68sopVTP7Cg/EAbMBFaJyFFgEfDaYN5U9Yzcm1t15K6UUj0ZcPkBY0yVMSbWGJNujEkHNgBXGmM2D1KfvatldOSulFI982Xkngh8KCI7gU+w5tzfEJGfi8iVg9u9nrkcOueulFJ9saX8QJfnlwy8W33Tde5KKdW3EblI3Dty1/oySinVI1syVEXkLvfm2DtFZKWIjB2c7lr8XFa3deSulFI982Xk7slQnQPMBZaJyKIubbYBGcaY2cCLwO/s7WZnft4kJh25K6VUT2zJUDXGfGiMqXc/3IC1Hn7QeMsP6MhdKaV6ZNcG2R3dArxtR+d640li0pruSinVM7s2yAZARD4PZAD39XLcnvIDDh25K6VUX+zaIBsRWQr8ECuBqamX19tbfkDn3JVSqke2bJAtIvOAf2AF9uLB6GhH3vIDGtyVUqpHdm2QfR8QCrwgIgA5xphBy171lh/QaRmllOqRXRtkL7W5X30KcDkBKK3tcfZHKaVOeyMyQzU+PID5aZE8siaLmsaW4e6OUkqdckZkcBcR/u+KGZTWNvH4x0eHuztKKXXKsav8QICI/EdEMkVko4ikD0ZnO5qTGsnUhDB25VUN9lsppdSIY1f5gVuACmPMROCPWHupDrrEiEAKqhqG4q2UUmpEsWuD7KuAJ92/vwhcJO5lM4MpISKIwqpB349bKaVGHLvKDyQDxwCMMa1AFRBjZ0d7khQRSFldM40tbd7nimsadfs9pdRpz9byA/2xq/yAR2JkEIB39N7U2sZF93/EP9dmDfjcSik1ktlVfiAPSAUQERcQAZT18Hpbyg94JEYEAlDgDu6HimqpaWxlX0H1gM+tlFIjmS3lB4DXgJvdv18HfGCMGfT00ePB3bqpujffCuo55fW9vkYppU4HdpUfeAx4WkQygXLghkHrcceORVjTMp6R+94CDe5KKQX2lR9oBK63t2v9C/J3EhnsR35l55F7aW0ztU2thAb48t2llFKjz4jMUO0o0b0csr3dsLegmthQfwCO6ehdKXUa82XOPVVEPnRvgL1HRO7soU2EiLzeIYv1y4PT3e6SIgLJrWjgWEU9tU2tXDIjAYDsMg3uSqnTly8j91bg28aY6cAi4HYRmd6lze3AXncW6xLgfhHxt7WnvZiVEsHB4hpW7CkC4Mo5SYCO3JVSpzdfMlQLjDFb3b/XAPuwkpY6NQPC3FmpoVg3VVtt7muPzp4YizHw4KpMxoQFcOa4aMIDXWSX1w3F2yul1CnphObc3QXB5gFdM1T/BkwD8oFdwJ3GmCFJE52TEkmwv5OK+hbOnRSHiDA+LpSPDpZQXtc8FF1QSqlTjs/BXURCgZeAbxpjumYJXQpsB5Kwiov9TUTCeziHrRmqAP4uBwvSowE4b3IsAD9YPo3i6iZue3qLLe+hlFIjja+1ZfywAvu/jTEv99Dky8DL7iJjmUAWMLVrI7szVD2WThtDsL+TcydZ51w4LpqvLZnApqPl1DYNyeyQUkqdUnxZLSNYSUr7jDF/6KVZDnCRu308MAU4Ylcn+3PTmWP5+HsXEh1y/B7u1IQwALJK6hiCZFmllDql+DJyPxv4AnChiGx3/ywXkdtE5DZ3m18AZ4nILmAl8D1jTOkg9bkbh0M6BXaACXGhAHxytJzZP1vBh/uLh6o7Sik17HzJUP0Y6LM2uzEmH7jErk7ZIS0mGIfAM5tyqGlsZUduJRdMHTPc3VJKqSEx4jNUexPgcpIaHUxmsbXPSF6F7tiklDp9jNrgDjA+NsT7e15lAyv2FPLAh5k6B6+UGvVsKT/gbrfEPR+/R0Q+sr+rJ84z7w6QX9nAo2uyuO/dA/zu3QOd2tU0tpBVqklPSqnRw5byA+567w8CVxpjZjAMFSJ7Mt4d3KcnhpNf2ci+wmpC/J08tOqwt4IkwF9WHuLTD67VEb1SatSwq/zA57DWuee4250SS1Mum5nA/140ievOSKG5rZ2axla+fsFEAv0cPLX+qLfdnvxqKupbqG7QNfFKqdHBrvIDk4EoEVklIltE5Iv2dG9gokL8ueviyaTHBnufWzQ+hmvmJfPK9jwq663yBJ6broXV1qYf/1ybxceHhmwlp1JK2c6u8gMu4AzgU1ilCH4sIpN7OIft5Qd8keTeSBtgSkIYn180lsaWdl7fkU91YwvFNU2AFdyNMfzunQP85YNDQ9Y/pZSym13lB3KBd40xde7kpdXAnK6NBqv8QH+S3cE9LTqY0AAXM5IimBIfxqvb872jdoCiqkaKa5poaGljW04F9c06TaOUGpnsKj/wKnCOiLhEJBg4E2tu/pQQFuhHRJCftyQBwJVzk9icXcFHB47/BVFY3ehdNdPSZtiUVT7kfVVKKTvYUn7AGLMPeAfYCWwCHjXG7B60Xp+Ee6+dzZ1LJ3kfezb1eOzjLPxdDiKD/SisbiS77PiSyLWZOu+ulBqZbCk/4G53H3CfHZ0aDMtmJnR6nBodzP87bzz/WH2EqQlhiAhFVY0cDfLDzynMS41i3eGyYeqtUkoNTL/BfTT7/mVTCfBzkhAeyHt7CymsbsTf5SA1OpiM9CgeXn2ExpY2Av2cw91VpZQ6Iad1cBcR7rrYWtSzK6+SXXlVtBtIjwlhVnIEre2G/YU1zE2NHOaeKqXUibGt/IC77QIRaRWR6+zt5uCLDw+ktLaZzOIaxsYEMyslAoBdeVXD3DOllDpxvozcPeUHtopIGLBFRN4zxuzt2EhEnMC9wIpB6OegSwgPBKxVMuPjQkmODCIq2I/dub4Fd2MMLW0Gf9eorsWmlBoh7Co/AHAH1lr4U6L0wIk6b3Icy2cl8IPlU7l2fjIiwszkCDZnl/P0hmz++N5B9hdWU17XzJGS2m6vf31nAQt+9b6ujVdKnRJOaM69t/IDIpIMXANcACzo4/W3ArcCpKWlnVhPB1lSZBAP3nRGp+dmJUew5lApP37FWtX5j9WHCXA5aWs3bPnxUgJcx2+0HiispqqhheyyeqYldtsbXCmlhpTPwb2f8gN/wtpar93KeeqZMeZh4GGAjIyMU74E4xcWjyU00MUl0xMID3Rx1/M7yK9q4EhJHdtzKjlzfIy3bXmdVacmt6JBg7tSatj5FNx9KD+QATznDuyxwHIRaTXGvGJbT4dBYkQQX18y0fv4X185k6r6Fub+YgXrj5R1Cu5ltVZwP1ZeP+T9VEqprvoN7r6UHzDGjOvQ/gngjZEe2HsTEezHjKRwNhzpnODkGbkfq9DgrpQafr6M3D3lB3aJyHb3cz8A0gCMMX8fpL6dshaNi+GpDdmsP1zG4ZJaFo2PPh7cy3WvVqXU8LOt/ECH9l8aSIdGgk/NTuTJ9Ue58ZENAFwxJ4ky75y7jtyVUsPvtM5QPVnz0qLYcM9FbM2p5G8fZnKkpJaqhhbAuqFqjKGvG8tKKTXYbMlQFZGbRGSniOwSkXUi0q2W+2gTExrAxdPjmZEUzoHCGgDGxYZQ29RKZX3LMPdOKXW6s2WDbCALON8YMwv4Be7ljqeDtOhgWtutVZ1z3CUL9KaqUmq42ZKhaoxZZ4ypcD/cAKTY3dFTVWrU8f1ZzxgbBcCholp+985+7nxum/dGq1JKDSVbMlS7uAV4++S7NLKkRR8P7hnp0QT5OdmZW8nL2/KoaWzlQGEN73zzvGHsoVLqdGRXhqqnzQVYwf2cXo6fsuUHTlbH4B4XFsDM5HDe2FlATWMrE+JC2F9YQ11TKyEBeu9aKTV07NogGxGZDTwKXGWM6XELo+HaIHswRQT7ER7oQgSigv2ZlRzpXRZ55Rxr9qqgSte+K6WGli0bZItIGvAy8AVjzEF7u3jqS40OJjLID6dDmO2+qRob6s/iCVZ5grzKxm6vOVZeT8Yv3+NgUc2Q9lUpdXqwK0P1J0AM8KB7fXerMSbD/u6emqYmhONyWt+Tnk0+5qdFkRwVBEB+ZfeR+ydHyymtbWb7sUomx4cNXWeVUqcFWzJUjTFfAb5iV6dGmp9eOZ3m1nYAxsWEsGh8NFfNTSY+LACH9BzcPWvjezqmlFIDpXf5bBAW6Of93eEQnrt1sfdxQnggeT0F9yIN7kqpwaN7wg2ypMggbwB/ZPUR1hwqAY6P3HsK/EopNVB2lR8QEfmLiGS6yxDMH5zujjxWcG8ks7iGX721j6fXZ1PV0EJBlXWTNb+Hm61KKTVQdpUfuAyY5P65FXjI1l6OYEmRQRRUNfDYx1kAZJfVe1fITBoTSl6lVWhMKaXsZNcG2VcBTxnLBiBSRBJt7+0IlBwZSEub4fnNuTgEssvr2F9g5YAtmRJHc2u7d128UkrZ5YTm3PsoP5AMHOvwOJfuXwCIyK0isllENpeUlJxYT0eoheNimBAXwpVzkvjGhZNobGln5f5iIoP9yEiPBiCzuJbGlrZh7qlSajTxObj7Un6gP6MxQ7U/UxLCWPntJfzxs3PJcBcW+/hQKbOSI0iOtNbBf/GxTSz/8xrK65ppbWvnvnf366YfSqkBsav8QB6Q2uFxivs51UF6TAgAre2GWckRpLiTnAyGvMoGbnt6C5uzK3jgw8M8uiZrOLuqlBrhbNkgG3gN+IaIPAecCVQZYwrs6+bokBQZiMshtLYbZqdEEBHkx2UzE7h0RgIV9c387PW9PLz6CADv7C7EGMOxigYe/9IC2toNT647ykXTxjDW/SWhlFK9sav8wFvAciATqAe+bH9XRz6X00FqdDBZpXXMSolERHjo82cAUF7XzC/f3McH+4vxdzoorG7kyfXZAGzNqeDfG3J4aWsue/Kruf8zo36jK6XUANlVfsAAt9vVqdFsbEwwVQ0tJEUEdno+OsSfsybEsOZQKV86O50n1h4lOsSf2qZW/ueJT6isbyEmxJ+1maW6R6tSql9afmCI3XXxZMpqm3sMzlfPTWbNoVKumJ3EjKRwUqKCeW17Hk+uz+b2CyaQHBnMD/67i8MldUwcE+p93ft7i0iJDmJqQvhQXopS6hTmy5z748DlQLExZmYPxyOAf2FN07iA3xtj/ml3R0eL2SmRvR779PxkpieFMy0x3FtdclpiGBdOi+e8SbHkVlilCj4+VOIN7m3thjuf28aSKWN44CZNDFZKWXxZLfMEsKyP47cDe40xc4AlwP0i4j/wrp1+RIRpiZ1H38H+Ls6fHIeIkBodzNiYYNYePr4XSlZpLXXNbWSX1w11d5VSpzBfMlRXA+V9NQHC3KtqQt1tW+3pnupqfloUO3MrvY935lYBkFOm6+KVUsfZURXyb8A0IB/YBdxpjGm34byqBzOTIyiqbqK4xio45gnu1Y2tVNW3AGCM4fK/ruGlLblU1DWzO69q2PqrlBoedgT3S4HtQBIwF/ibiPR4Z+90LD9gt1nJ1ly8J2DvyqvC4b43uyO3kjd25lNR38LuvGre3l3I71cc4JoH11JYpdUnlTqd2BHcvwy87C4alglkAVN7ang6lh+w2/SkcERgV241LW3t7Mmv4sxx1l6tP3xlF994Zps38G/LqWDd4TJa2gz/XKsZr0qdTuwI7jnARQAiEg9MAY7YcF7Vg9AAF+NiQ9iRW8mPX9lNY0s7nzszDYBj5dZqmi3ZFQCU1TWTVVpHkJ+TZzbmUNukt0KUOl34slnHs8B6YIqI5IrILSJym4jc5m7yC+AsEdkFrAS+Z4wpHbwuq1nJEXywv5jnPjnGHRdO5Io5ScSGHl+gtDWnolP7O5dOoqap1Rv029oNrW16W0Sp0cyXDNUb+zmeD1xiW49Uv/7n7HFEh/izeHwMF0+PByA1OpjSWqsu/PYcazVNsL8Tp0O4YUEqv317P7vzqjh/chw/eXU3+wtreOlrZ/X5PtlldYQH+hEVoitblRppNEN1BJqTGsmc1M7JUBljo/B3OthbUE1NYysxIf6cNzmOAJeDyGB/0mOC2ZVbRXu74e3dhZTXNVNW20RMaID3HI0tbVTUN5MY4S5F/PgmFqZHc9/1WstGqZFGN8geJX74qek8d+sib434MeGB/PGzc/nttbMBawnlrrwq9uRXU+7e+WlTlpW+UN/cijGG+1ccYPmf19DunrY5Vl7P/sIa2toNZbVNw3NhSqmT4suc++MiUiwiu/tos0REtrs30P7I3i4qX4mIN7gnhAd0OjYrOYK8ygZe22GV2fd3OdiYVc62nAoW/PJ9Hlx1mA8PlFBR30JeZQMltU20GzhcUstzn+Sw+LcfkFlcM+TXpJQ6OQMuPyAikcCDwJXGmBnA9fZ0TZ2MZPcGIAldqk561sf/e2MO0xLDWZgezTu7C/mfJz6hrrmNf649SmZxLQCZJbUUuNfF1ze38d+teTS3tvObt/YP4ZUopQbCjvIDn8Na557jbl9sU9/USUhyj9zjwzsH99mpkYyPCyEq2J+bF4/lrIkxFFY3khgRxDcumEhph2mXw8W1nZKeNmdXEOTnZOX+YnYcq0Qpdeqz44bqZMBPRFYBYcCfjTFP2XBedRKSewnuoQEuPvj2Eu/jxpY2FqZHMz8tiprGVv6x+jAhAS4Ea8PuriWJb1yYxuNrs9iZV9XtZq5S6tRjR3B3AWdgJTIFAetFZIMx5mDXhiJyK3ArQFpamg1vrbqaFG+VAh4f2/dWfIF+TjLSowGICPbji4vTCfJzsimrnMziWsICXQT6OfBzOqhpbGXZzASe3ZTDkZLaQb8GpdTA2bFaJhd41xhT505eWg30uHZOyw8MvqkJ4Xx09xLOHB9zQq/78eXT+c6lU5gwJtQ7554YEcSEuFBcDmFWcgTjYkPIKtXSwkqNBHYE91eBc0TEJSLBWBtk77PhvOokDWQD7YljQqmsb2FvfjXx4QFcPD2ey2YlEuTvZHxcCEdKeg7uLW3ttLWbHo81trRRXK2Fy5QaSgMuP2CM2Qe8A+wENgGPGmN6XTapTm0ZY6MAOFJaR2JEELdfMJG/3jgPsKZ6civqaWpto6m1jbzKBqztc+HqB9byizf29njO+1ccYPlfPva2VUoNvgGXH3C3uQ+4z5YeqWE1JzWSReOj2XCkvNtyyvFxobQba2OQF7bk8vDqI0xPDOdPN8xlT3412WX1XDs/hTd25fO/F04iJMCFMYZ39hRSWttEXXMboQGd/5Mrr2vmm//Zzm8+Pct7M1gpNXCaoaq6uePCSQCkRHUOtuPjrOmewyV1vLmzgLToYPYWVPPrt6xZuNqmVm54eD3/+OgIX/7nJzS2tHG4pNZbrbKkpnuW65pDJaw+WMLaQ1prTik72ZKh6m63QERaReQ6+7qnhsPZE2N5+paFXDMvudPz49wrcF7YfIy8ygZuv2ACSRGBrDpQQoDLQUJ4IHXNbXwmI4VNR8t5d08hH+w/nvbQU3Df5d5J6ojeqFXKVnZskI2IOIF7gRU29EmdAs6dFEewf+cplLBAPy6ZHs/K/cWIwIVT41nqrko5JyWSuy+dwm3nT+DX18wi2N/J1uwKVu4r9k7FeLYG7Gine2ORrFJdYqmUnezIUAW4A3gJ0OzUUe63184mITyQBWOjiQsL4KJpVnA/Iz2Ka89I4fuXTcXldDAnJZI1h0rZnF3BFXOSgO4j97Z2wx5vcNeRu1J2GvCcu4gkA9cADw28O+pUFx3iz2t3nM1Dn58PwOLxMXx+URrXnZHSqd28tEiOlNbR1m749Pxk/JziDe7Hyuupbmwhq7SWuuY24sICOFpW3+tSSqXUibMjQ/VPWLsvtXdNWe9KM1RHhzFhx1fR+Lsc/PLqWd3azE+zllRGBfsxPy2K2NAASmqaqG1q5fK/fkxyZJB3Tv+K2Uk8vjaL/MoGUqODh+YilBrl7FgtkwE8JyJHgeuAB0Xk6p4aaobq6WNemlV/5vzJcTgdQlxYACW1Tbyw+RhVDS3sLajmV2/tY0F6FEunjwFOfGrmuU05XPfQOtv7rtRoMOCRuzFmnOd3EXkCeMMY88pAz6tGtpjQAH7z6VksHGfVr4kLDSCvsoHH12ZxxtgopiSEsSe/mke+mEFzq7Wf67acSs6b7PuX/vv7iticXUFVQwsRQX6Dch1KjVT9Bnd3huoSIFZEcoH/A/wAjDF/H9TeqRHtxoXHp97iwgJY6V4W+b1lU7l8dhLGGEQEYwyzUyL44/sHqW9u5Z7l03o9Z2NLG5f/9WNuOWcce/KrAWsOP8Jdr/6Fzcc4UlrH95ZNHcQrU+rUZ0uGaoe2XxpQb9SoFRdm7QwVEeTn3dTbc49GRHjhtsXc89IuHllzhM8uSGV8XGiP53ljZwGZxbU8ue6od0OR3Ip6ZiZH0NZu+P2KAxRVN3HZzARmpwxeaeK/f3SYcyfFMiMpYtDeQ6mB0AxVNSTGuIP7FXMSCXA5ux0PcDm5Z/k0/F0OvvfSTv70/kFa2to7tTHG8OS6owDsLzy+5Z8nA3bNoRKKqpsQgb+szBykK4Hm1nZ++/Z+XtySO2jvodRAaXBXQ2JcbCgOgc9m9L5KKi4sgFvPHc8nRyv40/uH2JJd4T1W3djCXc/vYFdeFZfOiPc+7+9ycKyinsaWNp5en01UsB9fO38C7+8roqi6kQ/2F/E/T3zCI6uPdHqvX725l5e35pJZXMsv39jb7YukL5X11gbjxT1k3CrVh4RcAAAbjElEQVR1qhhw+QERuUlEdorILhFZJyI91nJXp7ezJ8aw/p6LmJXS9zTGXZdMYfXdFwBwqKiGH/x3F//emM1ja7J4ZXseX1sygfs/M5cAl4PEiEAmjQllb341y/60mpX7i/nC4nQumZEAwMp9xXz1qS18sL+Yl7YeH2XXNbXy+NqjPLHuKP/akM2jH2d5/yLwRbk7uPdUTkGpU4Uvq2WeAP4G9LZ1XhZwvjGmQkQuAx7GqumulJeIdNv6rzep0UGEBbrYllPJK9vzSI+19n6dnRLpvVF6zbxkAv2cFFY18s6eQgD+9rl5fGpWIi1thgCXg4c+yqSt3TAnJYLM4lrvDdwdxyppazfszquiqqEFgD+/f4ir5yUTGxrQb/8q6qzXaHBXp7IBlx8wxqwzxnj+ft4ApPTWVilfiAiT48N4e3ch7QaOlNSxNaeCcyYe313qt9fO5qdXziA12qpcOT42hE/NSkRE8Hc5mJ0SwbHyBsICXFw+O4m65jYq6q2g7JnuaTeQXVbP1XOTqGlq5ZVteT71r0JH7moEsHvO/RbgbZvPqU5Dk+PDaGhp8z42Bs6Z2H0NvCej9dPzkztt6j3fvenIogkxpLurWeaU1wOwObuCtOhg/JxW+y8sTmdaYjjv7C70qW/ldVZwr21qpb659UQvTakhYVtwF5ELsIL79/poc6uIbBaRzSUlJXa9tRqFprg3+p44JpSpCWEE+jmYP7b70sZF42OYnhjOdWekdnr+DHf5g3MnxXpH98fK62lvN2zNqeDsiTHMTokk2N/J7JQIls1IYEtOhXc7wHWHS7nw/lUUVDV0e88Kd3AHKK62Ru91Ta1c/cBaNh/tr8aeUkPDluAuIrOBR4GrjDFlvbXT8gPKV5PjwwCYnxbJd5dN4Uefmt7jEsrJ8WG8dee53XaNOm9yHHdeNImr5yWTGmWN7nPK68kqq6OmsZV5qVF855Ip/PLqmfg5HVw2KwFj4O3dhdQ2tXL3Czs5UlLHK9vyu72n54YqQEmtFdw3ZpWx/Vglnxy1viAOl2gJYzW8Blx+QETSgJeBLxhjDg68S0rB9KRwQgNcXDBlDBdOje//BV0E+jn51sWTvY9jQvzJrahnf0GN9/wzk4+v3Jk0JpRZyRE8se4oB4pqyK9qICkikNd35PO1JRM6nbvSPXcPx0fu6zKtMU1ZbRO/eXs/m7LKWfv9C0+430rZZcAbZAM/AWKwCoZtF5HNg9hfdZqIDPZny4+XctmsRFvOlxIdzLHyBvYXVuMQa7qnIxHha0smkFVaxzMbc/jyWeO45dzx7C2o7jYKL69r9u73eqi4hk+OlrP+iDu41zWTW1FPXmWDd4rHo6CqgdUHdTpSDQ1fVsvcaIxJNMb4GWNSjDGPGWP+7qkrY4z5ijEmyhgz1/2TMfjdVqeDnqZhTlZqVBA55fXsL6xhfFwogX7dz33pjAQmjQllbEww37l0MpfPTsTpEP7zybFO7SrqmxkfF4LTIfzp/UNc//f13jo3ZXXN3lU0O91bCHo8uiaLL/1zE6W1uspGDT7NUFWnhSnxYRyrqGdTVjlTE8J6bON0WDVuXr/jHIL9XcSHB7J8ViLPbMyhurGFbTkV3PrUZoqqG4kNDSA21B84XjcnNjSAstomb+aqZwtBj7yKBtoNvLItj/99dht78jsfV8pOdmzWodQp7/qMVP688hBVDS1MSwzvtV1ksH+nx7eeO57Xd+Tz3KYcCquaWLG3CICoYH/iwwNpam3n3W+eR3ZZHc9uyuHt3YXUN1tLOHfmVnY6l2flzb3v7KelzRAd4s+uhCqOltXz/cu0iqWyly8lfx8HLgeKjTEzezguwJ+B5UA98CVjzFa7O6rUQCREBHLFnCT+uy2PaYk9j9x7MislglnJEby/txg6bDQWHeJnBWRjbT0YHeLPir1F1DRa696D/Z1sy6nk9+8eIKusji+fle6tYtnSZm0n+HFmKe/tLaKyvpnvLZvSaZ1+V02tbbgcDpyOvnc7U8rDjvIDlwGT3D9nYu2lquUH1CnnjgsnUtPYSkZ69Am97sxx0Ty1IRu/DoE1MtifsybEdmoXE3J81H/57ESe35zLg6us6pQBLgcltU1cMy+ZmsZWxsUG88iaLG/7ouqmbss5O7ruofUsnhDDD/qoda9URwMuPwBcBTxlLBuASBGxZ4mDUjYaHxfKozdnEB54Yrs2LRwXTXNrO3XNbd6dpaK6TN8AxIQef+6Wc8az+2eXcvCXl7FwXDRrM0sxxvqiePTmDK6am9zptX2ti69ubGFXXlWPc/TNre00tbaxcl8RV/3tY2/27MkyRjcpHy3suKGaDHRcTpDrfq4bzVBVI9GCDiP97146hVvOGcc5E2O7tYsJOV50bExYAKEBLlxOB5PGhFHkXg/vGZ1PTwwnNtSfmcnW/H9fwd2zNj+/srHbsS/9cxPzf/4etzy5mR25VewrqD6JK7QcK69n+k/e7VRqWY1cQ7paRjNU1UgUFeLPpDGh+DsdzE6J5MeXTyciuPvo3zNy93MKkR2OT44/vqY+McJaH+9wCM98dRGP37yA0AAXh4t7D+6egJ1X2dBpZL07r4p1h8uYnBDGFHdGb2FV9y+Arg4V1XDpH1d3K3y2ObuchpY23t5V0O851KnPjuCeB3Qs7JHifk6pUeOGhWlce0YK/q7e/5fxlAuOCw3odHN0UvzxG7iJkcfn1SfHhzEmPJAJcSEcLqnr9bx73Wvom1vbKesw7fLU+qME+Tl54ssLeeX2swEorG6ksaWt2+YjFXXNtLqf25JdwYGimm51cDy7W605VOp9bl9BNWf9ZiVF1f1/aahTix3B/TXgi2JZBFQZY/SrX40qt5wzjt98elafbTzz8J517x6eOjkh/k7CArqvYZgQF8qRPqZl9hZU43LfzC1wT80UVzfy6vZ8rp6XTESQH0H+TiKD/SioauDah9bxnRd2eF/f3m5Y+oeP+OP7VnUQzxTRvg5bFQIccD8+UFTjDebbcirJr2pkx7HOyzrVqc+O8gNvAUeATOAR4OuD1lulTmH+LgcRQX7EhXVe9RId4k9MiD8JEYE9LnecMCaU/KpGHlp12Du6Bmud/KJfr2RvQTVnjrfm/fMqG2hvNzzwobURyW3nj/e2TwgPJLusnr0F1by6PZ/9hdaIv7y+mbK6Zv7zyTFa2topqrECd9f5+QOFNd4Er1UHigEodK/Nzyq1/rJoazcntCWhGj79LoU0xtzYz3ED3G5bj5Qawa6Yk8iMpO5bCS6eEIO/s+ex1A0LUtl8tJx739mPyyF89TwrYP97Qw6VDc1Mjg/j5sXprM0s4x+rD3P7M1tpN4YbFqQyNibEe56EiEA2HCnDMy3/x/cO8o8vZHjn4Utrm/noQIm35o0n+ANU1bdQUNXIFxaPpXVrHve9e5AlU8Z41+YfLbOC+09f28POvCpedU8DqVOXZqgqZaNfXt3z1M1fb5zXa5JSTGgA//zyQr7w2EYeXJXJmsxSwgJdrD5YwvJZifzhM3MxxhDo52BbTiUJ4YGcPzmOby2d3Ok8iRGBNLZYo+or5iTx+o583tldgNNhfak4BP67Lc87LXOsvIGaxhbCAv04UGRNyUxLCOfBm+K5+oG1/Oz1Pd6krKzSOtraDa/vzKeyvoWi6kaft02sqGvmzv9s51dXz/RurqIGn09z7iKyTEQOiEimiHy/h+NpIvKhiGxzb5a93P6uKjVy9ZV96vHdS6dSUd/ChiNlvLmzgJrGVu96eBEhyb3S5nNnpnHvdbMZ0yW4JoS7V+II3HvtLGYmh/PD/+4m2z3qXjgumu3HKimsbmSM+77AgcIaimsa+esHhwCYmhjG5Pgwls1IYGt2pXfknlVqbXXoKXe8/nCv2zZ08+GBYlYfLPFO9XiU1jZx+V/XsPGI7+dSvvNlzt0JPICViToduFFEpndp9iPgeWPMPOAG4EG7O6rUaDcrJYJ/fmkBK755Hl9fMoE5KRGcPeH4vrFJ7jLDV81N6vH1CRFWwE6PCSHY38XXl0ykrK6Zjw6W4HII50yMJa+ygdLaJi6cOgaHwE9e3cOyP61hU1Y5v7hqhnep5uSEMAqrG8kpq8fpEIqqm3h9Rz5+TiEs0MW6w6U99qEnm7KsVTkHizrfNP7dO/vZnVfNWz0svXxh8zHd8GSAfJmWWQhkGmOOAIjIc1hZqXs7tDGApxpTBNB9+xqlVL8umDoGgO8u615IbOm0McSHB3aaZ+8owR2YJ7hr1Xtujm7MKmdMWIC3YJox1hfJ+ZPj+PGru0mODOKPn53DxDHHl2x61s03t7UzLy2SbTmVvLA5lzPHxRAS4OTNnQWs2FvE/10xnWvmpfR5TZ7g7pn6AStp68UtuT2235Jdwd0v7mTx+BievXVRn+dWvfMluPeUgdq1dsxPgRUicgcQAiy1pXdKKa8vnT2uz+OJ7uxXz0YkY2NCCPRz0NjSTkJEIFM6lDqODwtk6fR4LpoWj8shOLoUJJvUIfHqrAkxbMuppN0YvnXxZLJK63hvbxHRof5854WdPLEum8ggP2YlRxAR5MeXz07nsY+zGB8XypzUCI6U1uHvdHCwqIY7nt1GgMvB7JQI2g0E+Tm9WxV6/OG9AwCsP2JtXTg3tfveuap/dt1QvRF4whhzv4gsBp4WkZnGmE5rpkTkVuBWgLS0NJveWikFkBYdzML0aJZOs0b/TocwaUwYu/KqSIwIIjkyiLAAFzVNrd4yCL0lZSVHBhHi76SuuY2zJ8ZyoLCGm84cyxljo5ifFsmlM+JxiPDjV3ZTUttEbkU9H7l3mZowJoT73zvIgvQoPrdwLACXzUrg1e35vL4jn6hgP28W75T4sE6ZsvsKqlmbWcY3l07i8Y+zeHj1YR686YzB/Nc2avkS3H3JQL0FWAZgjFkvIoFALNDpDoox5mHgYYCMjAytUKSUjQL9nDx/2+JOz01JsIK7Z439lIQwNmdXMCY8oJezWESESfFhbD9WSVp0MI/evKDTsTB38bU/fHau9/nqxhbm/GwF/9qQQ3NrO/sKatiaU4G/y8Gn56fw6nZrtraivoVVB0qYmhBGXFgguzrUvfdkzV47P4WaxlaeWn+U3Ip61h8u4+p5yfj1spxUdefLv6lPgEkiMk5E/LFumL7WpU0OcBGAiEwDAgGtDKbUMPPMu3umbKYmhuHnlE5FznozJT4MERgT5tuSx/BAayT+wX5rTFde18zKfUXMTApnRpI13++puVNQ1cjUhHDiQgM6jdy3HaskNtSflKggrpmXTEub4fq/r+fuF3fyUi9z9CfrSEktt/97K9WNLf223ZtfzaJfr+SBDzM7JZqdynwp+dsKfAN4F9iHtSpmj4j8XESudDf7NvBVEdkBPIu1YYeOzJUaZp55ds80zO0XTOSRL2b4tOnHLeeO49fXzOqznk5XZ4yN6vT4aFk9c1IjiQ0N4PozUvjtp2d5SylYI/cA6prbqGuy1tN75thFhBlJ4UwaE0pBVSNOh/Dox1m0t/ceVt7Ymc/dHcouPLcph1++sZc1h7qPM40x3PPyLt7cVcAnWX1VNLesOVRCYXUj9717gBe6fMnUuvt+qvHpUzPGvGWMmWyMmWCM+ZX7uZ8YY15z/77XGHO2MWaOe5PsFYPZaaWUbxaPj+GHy6dx0dR4wKpKuWTKGJ9eOzk+jBsXnti9sYz0KO/7esxJsW6I3nf9HJbNTPTW2pmaGO6tw7O/sIYt2RUcKaljXpp1DhErW/eMsVH84qqZZBbX8qNXd/dY+XJ3XhV3Pb+DF7bkUlLTRFltEz96ZTePrc3i5sc3sfpg5wD/6vZ8NrqD+v4ONXY+PFDMBb9fxdm//cBbsM3TJiE8kLiwAD7pUHAtu6yOuT9b0em5U4VmqCo1irmcDm85g6GQMTYaEVg6PZ6jZXUUVDUyp8tql9kpEewrrGZyfCjVDdaUyNf+tcW7sXjH1TGfyUjlMxmptLS1s/loOS9sPsbR0jqe+eoiGlva+OsHh2htMzyzMcf7mn0F1RwsqqG13fDy18/iBy/v4hvPbGXV3RcQHeJPY0sb9717gJnJ4VTUtXhr7BRUNfCt/2wnOsSfktomnt98jJ9eOcN7zmmJYThE2JV7fNOU/YXW++zKrepU9/9UoHcnlFK2SY0O5qWvncVNZ6YxNSGM8EAX6TGdSw7cdv4E/nzDPIL9Xd6Re3FNE0kRgYyPC+n2ZQDg53Twh8/O5ZtLJ7PucBmHS2r5cH8xD3x4mH+sPsLs1Ahe+H/WzeR9BdW8uCWXOamRzE+L4s83zKO6sZWn12cD8K8N2eRVNnDPZdOYlhjuHbn/ZeUhmlraeezmBVwwJY63dhXQ3m5obm3ncEktUxPDmZUSQWZJrXcaKbfCKqyWU17fqb++zOMPNlvKD7jbfEZE9orIHhF5xt5uKqVGivlpUQT6Obn70qn8pYeaOumxIVw5x8qy7Vge+adXzuCDby8htIeyyB7XZ6TgcgjPbsxh1YESwgJd7Pv5Mv79lUXMSY0kMSKQl7bmsr+whuvPsJKrpiSEcdHUMTy5/ihV9S38/aMjnD0xhrMnxjItMYys0joaW9rYV1DD/LGRjIsN4VOzkyiuaWJzdgWHS2ppaTNMTQhjdkoExljTQAC5FfXef+ZVNpBbUc+6w6XM+/l7fZZxHgq2lB8QkUnAPcDZxpgZwDcHoa9KqRFkelJ4v/P7UcH+OB2CyyEs7lBqoTdjwgK5dGYCz31yjJX7izhnYixB/s7j75kYzsGiWgJcDq6Yc7xMw21LJlBe18z1/1hHaW0TX18yEYCpCeG0tRsyi2s5UlJLujv796KpYwj0c/DilmPe6pnTE8OZlWz9VbHLG9yPj9zveGYrt/1rCx/uL6at3XTarrChuY11mb6XbLCDLyN3b/kBY0wz4Ck/0NFXgQeMMRUAxphilFKqH06HEBvqz/yxUd618/35ziVTaG5tp7S2mSVTOm/X6SmxsGxmAhFBx8+3ID2aW88bz8GiWmYkhXOW+4tkaqJ1c3fd4VKqG1sZF2sF95AAF9fOT+GVbfk8u+kYwf5OxsWGEBcWQHJkEB+7A7UnuGeX1bMzt4rdedW8vbsQgH0Fx2/U/nnlIT736Ebe31t0wv+OTpYvwd2XDbAnA5NFZK2IbBCRZT2dSDfIVkp19bMrZ/LD5dN8bj8uNoT/d/54/J2Obn8ZeObrP7sgtdvrvnvpFL5xwUR+dc0s71RRekwIYYEuXtqS5z23xy3njKOlvZ1NWeV8b9lUXO4EquvOSGHVgRKOlNSSW1FPgMtBU2s7re5lmp6A77lR29DcxnOfWDd8f/Lqbu98/WCz64aqC5gELMEqRfCIiHS7K6IbZCululo2M6HHm6h9+dbSyXz03SXdasovnTaGt/73XM6aENvtNS6ng+9cOqXTahynQ1iQHu0tatYxuI+PC+Xmxel8NiOVLy4e633+84vG4u908Kf3D1HT2Opd/gnWrlsAY2OC2VtQjTGGf2/MprK+he8tm0pBdSP3rzh4Qtd6snwJ7r6UH8gFXjPGtBhjsoCDWMFeKaVs53CItzxxRyLC9KTwHl7RO88SRqdDum0m8tMrZ3DvdbM73RSOCwvg0/OTeW2HVU7B80UyPjaE5bMS8Hc6+NzCNKoaWvjqU1v45Zv7WJAexW3nj+emM9N4Yl0WO3MHf09au8oPvII1akdEYrGmaY7Y2E+llBoUC8dZwT01Ksjn2jXfungywe4buWeOs9b2z0uL4u5LpvLi1xZ7M3Xf31fENy6YyL++ciYiwneXTSU2NIA3d3avYW83X/ZQbRURT/kBJ/C4p/wAsNmdpfoucImI7AXagLuNMbq9ilLqlDcrOYJAP0enKZn+xIcHctfFk7l/xUEmJ4Rx33VzmJsaSUSwH7ODI6lvtm7O3nRmGl8593gSWXigH2/ccU6nJaCDRYarBExGRobZvHnzsLy3Ukp19OymHNKigzl7Yve5+r7UN7cS7D+0if4issUYk9FfOy0/oJQ67Z1oDR2PoQ7sJ8K2DFV3u2tFxIhIv98qSimlBo9dG2QjImHAncBGuzuplFLqxNiVoQrwC+BeoHs9TqWUUkPKlgxVEZkPpBpj3rSxb0oppU7SgDNURcQB/AFrN6b+2mr5AaWUGgJ2ZKiGATOBVSJyFFgEvNbTTVUtP6CUUkNjwBmqxpgqY0ysMSbdGJMObACuNMboInallBomdm2QrZRS6hQybBmqIlICZJ/ky2OBoa18f2o4Ha9br/n0oNfsu7HGmH7ntYctuA+EiGz2Jf12tDkdr1uv+fSg12w/3SBbKaVGIQ3uSik1Co3U4P7wcHdgmJyO163XfHrQa7bZiJxzV0op1beROnJXSinVhxEX3H0tPzzSichREdklIttFZLP7uWgReU9EDrn/GdXfeU5lIvK4iBSLyO4Oz/V4jWL5i/tz3+muZzTi9HLNPxWRPPdnvV1Elnc4do/7mg+IyKXD0+uBEZFUEflQRPaKyB4RudP9/Kj9rPu45qH7rI0xI+YHa5u/w8B4wB/YAUwf7n4N0rUeBWK7PPc74Pvu378P3Dvc/RzgNZ4HzAd293eNwHLgbUCwSlxsHO7+23jNPwW+00Pb6e7/xgOAce7/9p3DfQ0ncc2JwHz372HAQfe1jdrPuo9rHrLPeqSN3H0tPzxaXQU86f79SeDqYezLgBljVgPlXZ7u7RqvAp4ylg1ApIgkDk1P7dPLNffmKuA5Y0yTMSYLyMT6f2BEMcYUGGO2un+vwcp0T2YUf9Z9XHNvbP+sR1pw77f88ChigBUiskVEbnU/F2+M8WybXgjED0/XBlVv1zjaP/tvuKcgHu8w3TbqrllE0oF5WJv6nBafdZdrhiH6rEdacD+dnGOMmY+1A9btInJex4PG+ltuVC91Oh2u0e0hYAIwFygA7h/e7gwOEQkFXgK+aYyp7nhstH7WPVzzkH3WIy2491d+eNQwxuS5/1kM/BfrT7Qiz5+n7n8WD18PB01v1zhqP3tjTJExps0Y0w48wvE/x0fNNYuIH1aQ+7cx5mX306P6s+7pmofysx5pwb3P8sOjhYiEuPekRURCgEuA3VjXerO72c3Aq8PTw0HV2zW+BnzRvZJiEVDV4U/6Ea3LfPI1WJ81WNd8g4gEiMg4YBKwaaj7N1AiIsBjwD5jzB86HBq1n3Vv1zykn/Vw31U+ibvQy7HuPB8Gfjjc/RmkaxyPded8B7DHc51ADLASOAS8D0QPd18HeJ3PYv1p2oI1x3hLb9eItXLiAffnvgvIGO7+23jNT7uvaaf7f/LEDu1/6L7mA8Blw93/k7zmc7CmXHYC290/y0fzZ93HNQ/ZZ60ZqkopNQqNtGkZpZRSPtDgrpRSo5AGd6WUGoU0uCul1CikwV0ppUYhDe5KKTUKaXBXSqlRSIO7UkqNQv8f7KGiLysT1S8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "def show_plot(points):\n",
    "    plt.figure()\n",
    "    fig, ax = plt.subplots()\n",
    "    loc = ticker.MultipleLocator(base=0.2) \n",
    "    ax.yaxis.set_major_locator(loc)\n",
    "    plt.plot(points)\n",
    "\n",
    "show_plot(plot_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 评估效果\n",
    "\n",
    "实际翻译的代码和训练类似，我们用上一个时刻的输出作为下一个时刻的输入，当遇到EOS时我们认为翻译结束。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(sentence, max_length=MAX_LENGTH):\n",
    "    input_variable = variable_from_sentence(input_lang, sentence)\n",
    "    input_length = input_variable.size()[0]\n",
    "    \n",
    "    encoder_hidden = encoder.init_hidden()\n",
    "    encoder_outputs, encoder_hidden = encoder(input_variable, encoder_hidden)\n",
    "\n",
    "    decoder_input = Variable(torch.LongTensor([[SOS_token]])) # SOS\n",
    "    decoder_context = Variable(torch.zeros(1, decoder.hidden_size))\n",
    "    if USE_CUDA:\n",
    "        decoder_input = decoder_input.cuda()\n",
    "        decoder_context = decoder_context.cuda()\n",
    "\n",
    "    decoder_hidden = encoder_hidden\n",
    "    \n",
    "    decoded_words = []\n",
    "    decoder_attentions = torch.zeros(max_length, max_length)\n",
    "    \n",
    "\n",
    "    for di in range(max_length):\n",
    "        decoder_output, decoder_context, decoder_hidden, decoder_attention = decoder(decoder_input, decoder_context, decoder_hidden, encoder_outputs)\n",
    "        decoder_attentions[di,:decoder_attention.size(2)] += decoder_attention.squeeze(0).squeeze(0).cpu().data\n",
    "\n",
    "        topv, topi = decoder_output.data.topk(1)\n",
    "        ni = topi.item()\n",
    "        if ni == EOS_token:\n",
    "            decoded_words.append('<EOS>')\n",
    "            break\n",
    "        else:\n",
    "            decoded_words.append(output_lang.index2word[ni])\n",
    "            \n",
    "        # Next input is chosen word\n",
    "        decoder_input = Variable(torch.LongTensor([[ni]]))\n",
    "        if USE_CUDA: decoder_input = decoder_input.cuda()\n",
    "    \n",
    "    return decoded_words, decoder_attentions[:di+1, :len(encoder_outputs)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以测试一下效果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> je suis juste curieuse .\n",
      "= i m just curious .\n",
      "< i m just curious . <EOS>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def evaluate_randomly():\n",
    "    pair = random.choice(pairs)\n",
    "    \n",
    "    output_words, decoder_attn = evaluate(pair[0])\n",
    "    output_sentence = ' '.join(output_words)\n",
    "    \n",
    "    print('>', pair[0])\n",
    "    print('=', pair[1])\n",
    "    print('<', output_sentence)\n",
    "    print('')\n",
    "\n",
    "evaluate_randomly()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 注意力的可视化\n",
    "\n",
    "注意力可以看出一种soft的对齐，我们可以观察模型怎么”对齐“源语言的词和目标语言的词。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_attention(input_sentence, output_words, attentions):\n",
    "    # Set up figure with colorbar\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    cax = ax.matshow(attentions.numpy(), cmap='bone')\n",
    "    fig.colorbar(cax)\n",
    "\n",
    "    # Set up axes\n",
    "    ax.set_xticklabels([''] + input_sentence.split(' ') + ['<EOS>'], rotation=90)\n",
    "    ax.set_yticklabels([''] + output_words)\n",
    "\n",
    "    # Show label at every tick\n",
    "    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "\n",
    "    plt.show()\n",
    "    plt.close()\n",
    "\n",
    "def evaluate_and_show_attention(input_sentence):\n",
    "    output_words, attentions = evaluate(input_sentence)\n",
    "    print('input =', input_sentence)\n",
    "    print('output =', ' '.join(output_words))\n",
    "    show_attention(input_sentence, output_words, attentions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input = elle a cinq ans de moins que moi .\n",
      "output = she s five years younger than me . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWIAAAEZCAYAAACtuS94AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAH7dJREFUeJzt3XmcHWWd7/HPl6Agy4AKbmwBB8WAbAkRB1RU4AZEmFEQEHRQNI4rLsjF5YIi3jvIqINXUCKC4y4iaC5GURAGFxQSEgIJorksQ1DEICCiLOn+zh9VDYemu89pUtV1lu+bV726TlWd53mqOfn1c556FtkmIiKas1bTBYiIGHQJxBERDUsgjohoWAJxRETDEogjIhqWQBwR0bAE4oiIhiUQR0Q0LIE4IqJhCcQRMSVU+K6k5zVdlm6TQBwRU2VfYDfgTU0XpNskEEfEVDmaIgi/UtLaTRemmyQQR0TtJG0CbG/7B8DFwD82XKSukkAcEVPhdcA3yv1zSPPEoyQQR8RUeCNFAMb2VcAzJW3RbJG6RwJx9D1Je0hav9w/UtKnJG3VdLkGhaSNgc/avq3l8LHAJg0VqesoE8NHv5O0FNgJ2BH4EnAW8BrbL2myXBEjUiOOQbDaRY3jIIqa2enAhg2XaSBIerOkbct9STpH0p8lLZW0S9Pl6xYJxDEI7pX0AeBI4PuS1gKe0HCZBsUxwM3l/uEU30q2Bt4LfKahMnWdBOIYBIcCDwBH274d2Bw4tdkiDYzVth8q9w8Avmz7TtsXA+s3WK6ukjbiiKiNpKuBVwB3AbcAL7O9rDx3ve0MdyY14hgAkl4l6beS7inbJ++V9OemyzUgTgAWUjRPzG8Jwi8BbmywXF0lNeLoe5JWAK+0fX3TZRlE5XDmDW3f1XJsfYr485fmStY9Mt47BsEfEoQb9RTg7ZK2L18vA86w/YcGy9RVUiPucZJ2nei87aunqizdStJpwDOA71I8tAPA9vmNFWpASNoD+DpF/+1F5eGZwD8DR9j+eUNF6yoJxD1O0i+BXYGlgIDnU3zg7wds+2UNFq8rSDpnjMO2/cYpL8yAKT+fb7W9eNTxnYEzbb+gmZJ1lzRN9L7fAW+2fS2ApB2Aj9g+uNlidQ/bb6g7j3LI9La2L5b0JGBt2/fWnW8P+LvRQRjA9hJJGVRTSiAGJO1J8Y/oHEmbAhvYvqnpcnXouSNBGMD2dXWugCDpycAWtpfWlUdVJB1n+xOS/i/wmK9+tt9VUT5vBuZStIU+m6Kf8ueBl1eQ9pTcQ40k6cmtD+rKg08hvbYeNvCBWNKJwCzguRSzQz0B+CqwR5PlmoSlks6iKDPAERTNFJWRdBlwIMXnZRFwh6Sf235vlfnUYOQB3cKa83k7MBv4FYDt30p6WkVpT9U91OXTwI8kHQuMPK+YCZxSngvSRoykJcAuwNW2dymPLbW9Y7Ml64ykdYG3Ai8uD10OfM72/RXmsdj2LpLeRFEbPrGXfkcjJG0AUHWXKUm/sv2Clt/T2hSfp8p/P3XdQ50kHQAcB2xPUatfDpxq+/81WrAuMvA1YuBB25ZkeLh/Y88oA+6nqbd2sbakZwKvAT5UYz61KNvNv0LRdCBJfwRePzK4oAL/KemDwJMk7QO8Dag0yEzBPdTG9oXAhU2Xo5uljQbOlXQmsHHZ1ncx8IWGy9Sxcq7dH0v6jaQbR7aKszkJuAhYYfsqSdsAv604jzrNA95reyvbWwLvo9r/x8cDfwSuBd4CLAA+XGH6UP891ELSuS37p4w696OpL1F3GvimCYCyFrMvRfevi2z/uOEidUzSr4H3ULTdDo0ct31nY4XqMpKusb1Tu2PdrFfvYaS5pty/2vauY50bdGmaAMrA2zPBd5R7ygUZa1P2JHkzMJ2Wz0xV/XAlPQf4HPB02ztI2hE40PbJVaQP3Cjpf1F8tYdiOszKvjVIuomxezRsU1Ue1HwPNZqoppdaYGlgA7Gkexn7gyCKzv5/N8VFerwulXQqcD6PHjVW5Yi67wE/pWi2GWpz7ePxBeD9wJkAtpdK+jpQVSB+I/BR4Dvl658CVfYtntWyvy5wCEVbbpXqvoe6rFdOAL8WRRv6LhT/xgQ8qdGSdZE0TfQ4SZeOcbjSEXWSltjeuar0xkj/Ktu7jfoaW1mekmZRPGScziOVD9fZ60PSItszK0xvyu+hCuN8Ph9m+6VTVZZuNsg14glrLLb/NFVlWRNT9EG+UNL+thfUlP4qSc+m/IYi6WDg9xWm/zWKxSqvA4YrTBd4zHwfa1HUkKv+t1XrPdQlgbYzA1sjbmnXE480Uaj86Yrb9yon6UjbX5U05qAK25+qMK97KVZTeAB4iIqbb8peGPOAf6CYQPwmiglhbqko/Z/Z3rOKtMZJ/1Ie+Qytpph7999s/6bCPGq9hzqVQ76fY/ualmNbAkOjVnYeWANbI7a9NUC5ftkRwNa2Tyo/IM+sMq9yWPC2FO2HI/lfvobJjvR3Hmu8fqV/XW1vWH6DeNQ9rKlRf0QWAJdS1CjvA14NVPXH5MRy9OEl1DP72oU88kedcv8ASSP5VHEfdd9DnVYD50va0fZ95bGzgA8CCcQMcCBucTrFV72XUfSXvZfigchuVSRejkY7hmL+gSXA7sAVZX6Pm+0zy91tgGNs313m92Tgk2uS9mjj3MMvWPO5FEb+iDyX4vf9PYpg9jrgyjVMu9UbgO0ohq+PfK03xQPOKszk0eV/JUX5q+xrXfc91Mb2Q5IuoBgQdE5Z2dnUdq8O266e7YHeKIaiAixuOXZNhelfS1GLXFK+3g44v8L0F3dyrMvv4XKKFRxGXm8IXF5h+jfU/BmqtfxTcQ91b+Vn5vJy/8PAu5ouUzdtGVkHD0maxiMPijal2och97uc90HSOrZ/TVEDrMpaZS2YMo+nUP03nbrv4enAgy2vHyyPVeUXkmZUmN5odZcf6r+HWpWfGZV9xg/jkf7QQZomAD4DXAA8TdLHgYOpdnjqSkkbU6wO8WNJI6vZVuWTwBWSvl2+PgT4eIXpQ/338GXgyvLrK8A/UqzoUJXdgSXlA9oHeORhY1Vdv+ouP9R/D48h6Rm2b68wyS9StA1f61HTYg66ge010UrSdhTtnQIucU3rm6lYuXYj4Ie2H2x3/STSncEjbc4/sb28qrTHyKuue9gVeFH58nKPMZn4GqS91VjHXVGvjDKP2spfpl/7PYyR5/dtv6LC9Naj6Jb4atsXV5VuP0ggjohoWNqIIyIalkA8iqS5ST/pd2v6U5FHr6dfN0lnS7pD0nXjnJekz0haIWmp2qy0DgnEY6n7Q5L0k36359Hr6dftS8CcCc7vRzH4aVuKe/1cuwQTiCMiJsHFqNiJ5qI5CPiyC7+kWHRiwtG6fd19TeXyR1P1vqSf9Ceb/i67tv3W+hhbbLklu86c2XEei6+e/Iyo3fQ7AlbZ3nRN8pszZ45XrVrV0bWLFi1aBrSu+TjP9rxJZLcZcGvL65XlsXEnsurrQBzR7X5+xRW157HeOpVNDzKmadOm1Zr+0NDqNe6it2rVKhYu7GxEtaT7bc9qf2V1EogjYiBMYVfd24AtWl5vTpvJjdJGHBF9z8DQ8HBHWwXmA68ve0/sTrGc2YTza6dGHBEDwLii2WElfQPYC9hE0krgRIpZ8bD9eYopXfcHVgB/pYMlrRKII6L/GYYrapmwfXib8wbePpk0E4gjYiB083QOCcQR0fcMDCcQR0Q0KzXiSZB0MzDLdme9ryMi2rBdVY+IWnRdII6IqEM314gb7UcsaX1J35d0jaTrJB1annqnpKslXVtO2j5y7dmSrpS0WNJBDRY9InqMO/yvCU0P6JgD/M72TrZ3AH5YHl9le1eKWYuOLY99iGL1idnAS4FTJa0/OkFJcyUtlJQVYiMCGHlY19nWhKYD8bXAPpJOkfQi2/eUx0eWCF8ETC/39wWOl7QEuIxiVeEtRydoe57tWVM9VjwiulunKyo3odE2Ytu/KSdN3h84WdIl5akHyp9DPFJGUax1dcMUFzMiel2XP6xruo34WcBfbX8VOBWYaE7AiyjajlW+d5cpKGJE9AGTGvFEnk/R1jsMPAS8FThvnGs/Bvw7sFTSWsBNwAFTUsqI6HkZ0DEO2xdR1HRbTW85v5Bicg1s/w14y1SVLSL6Szd3X2u6RhwRMQWa65rWiQTiiOh7brBrWicSiCNiIAx3ca+JBOKI6HuZfS0iogvkYV1ERJPs1IgjYmzrrbNO00VYY6tXP1Rr+uUYrjWWGnFERIMMDCUQR0Q0KzXiiIiGJRBHRDTIeVgXEdG81IgjIhqWQBwR0aCi10SGOEdENCqT/kRENKnB1Tc6kUAcEX1vZKmkbtX0Ks6TJml9Sd+XdI2k6yQd2nSZIqL7DZdd2NptTejFGvEc4He2XwEgaaOGyxMRPSA14mpdC+wj6RRJL7J9T+tJSXMlLZS0sKHyRUSXsc3Q8HBHWxN6LhDb/g2wK0VAPlnSCaPOz7M9y/asRgoYEV3JHf7XhJ5rmpD0LOBPtr8q6W7gTU2XKSK6Xzd3X+u5GjHwfOBKSUuAE4GTGy5PRHS5kV4TnWztSJoj6QZJKyQdP8b5LSVdKmmxpKWS9m+XZs/ViG1fBFzUdDkiordU8bBO0jTgdGAfYCVwlaT5tpe3XPZh4Fzbn5M0A1gATJ8o3Z4LxBERk1Y+rKvAbGCF7RsBJH0TOAhoDcQG/q7c3wj4XbtEE4gjou9VOKBjM+DWltcrgReMuuYjwI8kvRNYH9i7XaK92EYcETFpkxjQsclIF9hymzvJrA4HvmR7c2B/4CuSJoy1qRFHxECYRNe0VRN0f70N2KLl9eblsVZHUww8w/YVktYFNgHuGC/D1IgjYiDYnW1tXAVsK2lrSU8EDgPmj7rmv4CXA0h6HrAu8MeJEk2NOCL6nqGSeSRsr5b0DoqeW9OAs20vk3QSsND2fOB9wBckvafM+ii3aaBOII6YQN3zE0iqNf2p0BP3UF2vCWwvoOiS1nrshJb95cAek0kzgTgi+l63T4OZQBwRAyGBOCKiYU3NNdyJBOKIGADNzazWiQTiiOh7HXZNa0wCcUQMhKYmfe9EAnFE9L2q+hHXJYE4IgZCN/eaaHSIs6R3Sbpe0l1jTbAcEVGJDieFbypYN10jfhuwt+2VDZcjIvpdasSPJenzwDbADyS9R9JnJW0k6ZaRKeMkrS/pVklPkPRsST+UtEjSTyVt11TZI6L3DA+5o60JjQVi2/9CMXP9S4G7ymP3AEuAl5SXHQBcZPshYB7wTtszgWOBM6a80BHRk4rua2mamIxvAYcCl1JMMXeGpA2AfwC+3TLByDpjvbmcxHmyEzlHRJ/r5od13RiI5wP/W9JTgJnATyiWG7nb9s7t3mx7HkXtGUnd+5uPiCnUXG23E103Mbztv1BMvnwacKHtIdt/Bm6SdAiACjs1Wc6I6C0edkdbE7ouEJe+BRxZ/hxxBHC0pGuAZRQrp0ZEtJU24gnYnl7ufqncRo6fB2jUtTdRrgMVETFZzhDniIhmdXETcQJxRAwAN9f+24kE4ogYCN3cayKBOCL6Xtasi4joAgnEERFNsvFQek1ERDQqNeKIiIZ1cRxOII6I/peHdRERTXMCcUREw8xwHtZFRDQrNeKIiAY5TRMREV0ggTgiolnu3ibiBOKIGAxpmoiIaJLNcCaGX3OSptkearocEdF7un1ARy1r1kk6SdK7W15/XNIxkt4v6SpJSyV9tOX8dyUtkrRM0tyW43+R9MlynboXSvpXScvL9/9bHWWPiD7k6hYPlTRH0g2SVkg6fpxrXlPGqmWSvt4uzboWDz0beH1ZoLWAw4DbgW2B2cDOwExJLy6vf6PtmcAs4F2SnloeXx/4le2dgOuBfwK2t70jcPJYGUuaK2mhpIX13FpE9KSiD1v7bQKSpgGnA/sBM4DDJc0Ydc22wAeAPWxvD7z7MQmNUksgtn0zcKekXYB9gcXAbi37VwPbUQRmKILvNcAvgS1ajg8B3yn37wHuB74o6VXAX8fJe57tWbZnVX1fEdGrOlvBuYPmi9nACts32n4Q+CaPXVH+zcDptu8CsH1Hu0TrbCM+CzgKeAZFDfnlwP+xfWbrRZL2AvYGXmj7r5IuA9YtT98/0i5se7Wk2WU6BwPvAF5WY/kjoo8Md75m3SajvlHPsz2v3N8MuLXl3ErgBaPe/xwAST8HpgEfsf3DiTKsMxBfAJwEPAF4LbAa+Jikr9n+i6TNgIeAjYC7yiC8HbD7WIlJ2gBYz/aC8gZvrLHsEdFHXLYRd2jVGn6jXpviW/1ewObA5ZKeb/vuid5QC9sPSroUuLus1f5I0vOAKyQB/AU4Evgh8C+SrgduoGieGMuGwPckrQsIeG9dZY+I/lNRr4nbKJpPR2xeHmu1kuLZ1kPATZJ+QxGYrxov0doCcfmQbnfgkJFjtk8DThvj8v3GSsP2Bi37v6don4mImLSKAvFVwLaStqYIwIdRfONv9V3gcOAcSZtQNFVM+A2+ru5rM4AVwCW2f1tHHhERnavmYZ3t1RTPpy6i6Ml1ru1lZZfdA8vLLqLorLAcuBR4v+07J0q3lhqx7eXANnWkHRExaRXOvmZ7AbBg1LETWvZN0XTacfNpz4ysi4h4vAx4qHtH1iUQR8RA6OYhzgnEEdH/Ohus0ZgE4ogYCJPoRzzlEogjJjBtWr3/RB5cvbrW9AGeuHb+mUOaJiIiGtXt02AmEEdE/7NxJoaPiGhW1qyLiGhYmiYiIppU4ci6OiQQR0Tfy8O6iIjGmeGh7m0kTiCOiP6XpomIiC6QQFwPSdNG1rSLiJhIF8fhiSeGLyc7fnfL649LOkbSqZKuk3StpEPLc3tJurDl2s9KOqrcv1nSRyVdXb5nu/L4ppJ+LGmZpLMk3VLOaI+kIyVdKWmJpDPLZayR9BdJnyxXfX5h1b+QiOg/Iw/rKljFuRbtVug4G3g9PLz00WEU6zHtDOxEsfryqZKe2UFeq2zvCnwOOLY8diLwE9vbA+cBW5Z5PQ84FNjD9s7AEHBE+Z71KdaD2sn2z0ZnImmupIWjVmGNiEFWLh7aydaECZsmbN8s6U5JuwBPBxYDewLfKJsE/iDpP4HdgD+3yev88uci4FXl/p7AP5V5/VDSXeXxlwMzgavKhUafBNxRnhsCvjNBmecB8wAkdfGXkYiYOma4x4c4nwUcBTyDooa8zzjXrebRNex1R51/oPw51EG+Av7D9gfGOHd/2oUjYrK6uddEJ4uHXgDMoaj1XgT8FDhU0jRJmwIvBq4EbgFmSFpH0sYUtdp2fg68BkDSvsCTy+OXAAdLelp57imStur8tiIiRrE72xrQtkZs+0FJlwJ32x6SdAHFQ7JrKNrAj7N9O4Ckc4HrgJsomjHa+SjwDUmvA64Abgfutb1K0oeBH5Vt0w8Bb6cI9hERk2L3+MTwZSDcHTgEHl6h9P3l9ii2jwOOG+P49Jb9hcBe5ct7gP9he7WkFwK72X6gvO5bwLfGSGuDdmWOiBiti1smJg7EkmYAFwIX2P5tDflvCZxbBvsHgTfXkEdEDLweXrPO9nJgm7oyL4P7LnWlHxEBgOn5XhMRET3N9HgbcUREP+jZpomIiP7QXNe0TiQQR0T/yzSYEb1reLjeQZxPXLv3/wnWHeDKaQ7W2PBQAnFERGOyVFJERNPSNBER0bQeHtAREdEvEogjIhrWzQM6OpkGMyKip43MvlbFCh2S5ki6QdIKScdPcN2rJVnSrHZpJhBHxECoYs26cu3M04H9gBnA4eXkaKOv2xA4BvhVJ2VLII6IAdBZEO6gHXk2sML2jbYfBL4JHDTGdR8DTgHu76R0UxaIJW0s6W3l/qNWfI6IqFV1TRObAbe2vF5ZHnuYpF2BLWx/v9PiTWWNeGPgbVOYX0TEwyZRI95kZCX4cpvbaR7l3OqfAt43mbJNZa+JfwWeLWkJxdJH90k6D9iBYmXnI21b0gnAKylWbv4F8Jby+GUU7S0vpQjqR9v+6RSWPyJ61CRH1q2yPd4DttuALVpeb14eG7EhRUy7rBya/QxgvqQDy9WJxjSVNeLjgf9ve2eKZZZ2Ad5N0eC9DbBHed1nbe9meweKYHxASxpr255dvu/EsTKRNHfkL1lN9xERPcd4eLijrY2rgG0lbS3picBhwPyHc7Hvsb2J7enlEnG/BCYMwtDsw7orba+0PQwsAaaXx18q6VeSrgVeBmzf8p7zy5+LWq5/FNvzbM+a4C9aRAwag4c72yZMxl4NvINiRfvrgXNtL5N0kqQDH2/xmhzQ8UDL/hCwtqR1gTOAWbZvlfQRYN0x3jNEBqNExCRUNbLO9gJgwahjJ4xz7V6dpDmVNeJ7KdpPJjISdFdJ2gA4uN4iRcSgqKj7Wi2mrFZp+05JP5d0HfA34A9jXHO3pC8A1wG3U7THRESskUyD2cL2a8c5/o6W/Q8DHx7jmr1a9lcxThtxRMRj2AwPZRXniIhmpUYcEdEsk0AcEdEYZ4WOiIimGbfrJNygBOKIGAipEUdENGy4/fDlxiQQR0TfKwZrJBBHRDQrTRMREc1K97WIiIblYV1ERKPM8PBQ04UYVwJxRPS9DOiIiOgCCcQREQ1LII6IaJTTfS0iomkmAzoiIhpjd/cQ5yZXcX4USdMl/VrSlyT9RtLXJO1dLq/0W0mzJa0v6WxJV0paLOmgpssdEb2gs/Xq+n7Nug79PXAI8EaK9epeC+wJHAh8EFgO/MT2GyVtDFwp6WLb940kIGkuMHfKSx4RXS1zTXTuJtvXAkhaBlxi25KupVijbnPgQEnHltevC2wJXD+SgO15wLwyje5tnY+IKZVeE517oGV/uOX1MEVZh4BX275hqgsWEb2tmwNx17QRd+gi4J2SBCBpl4bLExG9wO58a0C31Yjb+Rjw78BSSWsBNwEHNFukiOh2BoaduSbasn0zsEPL66PGOfeWqSxXRPSD5npEdKJrAnFERJ0SiCMiGpZAHBHRoOI5XPoRR0Q0yLiLhzgnEEfEQMiadRERDUsbcUREo5w24oiIJnX7mnW9NsQ5IuJxqWoaTElzJN0gaYWk48c4/15JyyUtlXSJpK3apZlAHBEDYXh4uKNtIpKmAacD+wEzgMMlzRh12WJglu0dgfOAT7QrWwJxRAwAg4c72yY2G1hh+0bbDwLfBB61QIXtS23/tXz5S4rpeyeUQBwRA8Ed/gdsImlhy9a60MRmwK0tr1eWx8ZzNPCDdmXLw7qI6HuTfFi3yvasNc1T0pHALOAl7a5NII6IgVBRr4nbgC1aXm9eHnsUSXsDHwJeYvuB0edHSyCOiAFQWT/iq4BtJW1NEYAPo1hb82HlghVnAnNs39FJognEETEQ2vWI6ITt1ZLeQbFa0DTgbNvLJJ0ELLQ9HzgV2AD4drmY0H/ZPnCidBOII6LvVTmgw/YCYMGoYye07O892TQTiCNiADS3Hl0nEogjYiCYzDUxZco+f3PbXhgRA6Wb55rou0Bsex4wD0BS9/7mI2IKuZKHdXXpu0AcETFaty+V1LNDnCUtkPSspssREb2hqtnX6tCzNWLb+zddhojoHWkjjohoVLqvRUQ0LouHRkQ0yIbh4aGmizGuBOKIGADNPYjrRAJxRAyEBOKIiIYlEEdENKybB3QkEEdE/3O6r0VENMrAcGrEERHNStNERESj0n0tIqJxCcQREQ2qcs26OiQQR8QAMM4Q54iIZnXzpD+1TAwv6TJJN0haUm7ntZybK+nX5XalpD1bzh0gabGkayQtl/SWOsoXEYNnICaGl/RE4Am27ysPHWF74ahrDgDeAuxpe5WkXYHvSpoN3Emx1txs2yslrQNML9/3ZNt3VVXWiBg83dxGvMY1YknPk/RJ4AbgOW0u/5/A+22vArB9NfAfwNuBDSn+MNxZnnvA9g3l+w6VdJ2k90nadE3LHBGDpajtDne0NeFxBWJJ60t6g6SfAV8AlgM72l7cctnXWpomTi2PbQ8sGpXcQmB7238C5gO3SPqGpCMkrQVg+/PAfsB6wOWSzpM0Z+R8REQ7/dg08XtgKfAm278e55rHNE20Y/tNkp4P7A0cC+wDHFWeuxX4mKSTKYLy2RRB/MDWNCTNBeZOJt+I6H/Dw907su7x1igPBm4Dzpd0gqStOnzfcmDmqGMzgWUjL2xfa/vTFEH41a0Xlm3JZwCfAc4FPjA6A9vzbM+yPavTm4mIATAy8U+7rQGPKxDb/pHtQ4EXAfcA35N0saTpbd76CeAUSU8FkLQzRY33DEkbSNqr5dqdgVvK6/aVtBQ4GbgUmGH73baXERHRljHDHW1NWKNeE7bvBE4DTitrq609pr8m6W/l/irbe9ueL2kz4BeSDNwLHGn795I2BI6TdCbwN+A+ymYJigd4r7R9y5qUNyIGU7ePrFM3F25NlcE+ImpUdwyRtGhNmxrXWmua11nnSR1de//9961xfpOVkXURMRC6udKZQBwRA8AMZ66JiIjmdHsbcQZERMRgqKj7WjmY7AZJKyQdP8b5dSR9qzz/qw56kyUQR8QgcMf/TUTSNOB0ikFlM4DDJc0YddnRwF22/x74NHBKu9IlEEfEQKhoronZwArbN9p+EPgmcNCoaw6imEMH4Dzg5ZI0UaJpI46IgVDREOfNgFtbXq8EXjDeNbZXS7oHeCqwarxE+z0Qr6IcnTcJmzDBL6wCST/pd3sek0q/TWVvjdMHOp1CYSIXlfl2Yl1JrfPkzLM9r4IyjKuvA7HtSU+ZKWlhnZ25k37S7/Y8ej39sdieU1FStwFbtLzevDw21jUrJa0NbEQ5ve940kYcEdG5q4BtJW1dLoZxGMX0va3mA/9c7h8M/MRt+s71dY04IqJKZZvvOyiaOqYBZ9teJukkYKHt+cAXga9IWgH8iSJYTyiB+LFqbQtK+km/B/Lo9fRrZXsBsGDUsRNa9u8HDplMmn096U9ERC9IG3FERMMSiCMiGpZAHBHRsATiiIiGJRBHRDQsgTgiomEJxBERDftvTIeB3j4s3HIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate_and_show_attention(\"elle a cinq ans de moins que moi .\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input = elle est trop petit .\n",
      "output = she s too short . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUYAAAEZCAYAAADrD4zSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGMBJREFUeJzt3X+0H3V95/Hni6CiGEEJHoWAoBtWIz/DFewKK61AA6K4By0/bKsWjWvFY1fRQtsTXaTHUlpd7QI1KriiK7IsalZjw6pQbKtC+BVINJwcFAmiNEAhC8qPe1/7x8xN5k7uvd/vJd/MzPfm9ciZk+98Zr4z7wTuO5/5fD7z+cg2ERGxxU5tBxAR0TVJjBERNUmMERE1SYwRETVJjBERNUmMERE1SYwRETVJjBERNUmMERE1SYwRs5gKX5f0irZjGSZJjBGz2/HAq4B3th3IMElijJjdzqRIim+QtHPbwQyLJMaIWUrSPOCVtr8NfAd4U8shDY0kxojZ6w+Ar5SfLyOP031LYoyYvf6IIiFi+0bgxZL2aTek4ZDEGDELSdod+O+2760Unw3MaymkoaJMVBsRMVFqjBGzjKR3SVpQfpakyyQ9Imm1pMPajm8YJDFGzD7vB35Wfj4dOBjYH/gA8OmWYhoqSYwRs89Ttp8sP58EfNH2A7a/A+zaYlxDI4kxYvYZk/RiSbsAr6MYwzju2S3FNFQyEj5i9lkKrALmAMttrwGQ9FrgrjYDGxbplY6hJely23/Qq2xHVL7+N9f2Q5WyXSl+5v9fe5ENh9QYY5i9srojaQ5weEuxdM0LgPdKGv87WgNcbPtXLcY0NNLGGENH0rmSNgEHl8NQHin37we+0XJ4rZP0GuDGcveL5Qbwo/JY9JBH6ehL2ZD/x8BRgIF/Ai6x/ZsWY/q47XPbun9XSfoh8B7bt9TKDwU+Y/vIdiIbHkmM0RdJVwKbgC+VRWcAu9t+SwuxvNz2TyQtmuy47ZubjqlLJK21vXCmx2KLtDFWSDoKWGD7Mkl7As+1/dO24+qIA2s/UNdKWttSLB8AlgB/O8kxA7/TbDidI0nPr3a8lIUvIM1nfUliLEn6CDAC/HuKGUmeQVE7SptM4WZJr7b9QwBJR1IMCWmc7SXlxxPqj/LlI/+O7pPANZLOBsZrz4cDF5THooc8Spck3QocBtxs+7CybLXtg1uK51m2H+9V1mA8P6b4R+PnZdG+wDrgKcBt/D1Jutn2ol5lOyJJJwEfpui5N7AWuND2/2k1sCGRGuMWT9i2JMPmMV9t+gFQ/wGfrKwpi1u671YkvQjYG3h2OSmCykPPA57TWmAdYvubwDfbjmNYJTFucaWkzwC7S3oXxSSfn206iK7+0Nu+W9IhwNFl0fdt39ZSOL8LvB2YD3yiUv4I8GdtBNQlkq60/Xvl5wts/2nl2DW2j28vuuGQR+kKScdRrKomYKXt/9tCDG+j+KEfoRiLNp4YNwFfsH110zGVcb0feBcwfv//BCyz/XdtxFPGdIrt/93W/btK0i2V5qAJTQvVYzG1JMaO6toPvaTVwG/ZfrTc3xX4QVttsGUMLwL+EtjL9gmSFpYxfr6tmLqgmgwnSYxpg+3DDt91L2lT5e2J6rZJ0iMthjZf0vPKiUY/J+lmSW0+AgkYreyPsqU225bLgJXAXuX+ncCftBdOZzxH0mGSDqdskpG0aHy/7eCGwQ7fxmh7btsxTOGPbH9K0u8Ce1Cs+HY5cE1L8VxG8UrZ18r9NwFt18zm2b5S0rkAtp+SNNrrSzuA+9jS9vpLJrbD/rL5cIbPDp8Yy0GvU7L9YFOx1IzXxl5PMdHoGkmt1dBsf0LSdRSvBAK8o/7KWQselbQHxXAUJL0aeLjdkNpn+7fbjmHY7fBtjJJ+SvGDpfJ32JKUbPulLcV1GcUj4kuBQyjm1rvOduOzx5Sz1qyx/fKm7z2d8pXAv6MYq7cG2BN4s+3VrQbWAZKeDRxQHTkgaV9gtLZyYExih29jtL1/mfxeBnyEoud3f+C1FOtltOVM4J+Bb9p+DHg+LbWf2R4F1pU/WF2yFvgaRe/9ryiGV93ZakTd8RRwdW087ueAF7cUz1DZ4WuM4yRdAowBv2P7FZKeD1xj+1WJByRdT/Fm0A3Ao+Pltt/YRjxlTFdSjF38clnU2sQWXSTpbyhq+peV/6h9I0N1+rPDtzFWHGl7kaRbAGw/JOmZiWezXSgWVhonindv29SliS266HPAMoqOsz8sf48+JDFu8WTZljbekL8nRY0t8RR2tv2P1YKyHatNnZnYoovKqdkk6QDgNLa8tRQ9JDFu8WmK9qoXSvpL4M3AX+zo8Uh6D8UEtS8tB3mPm0vRBtqmw4F/kTRhYgtJt9PSxBZ1kl5ku80hMp+nqDneXp+GLKaWNsYKSS+nWG5SwHdt/3hHj0fSbhQdPx8Hzqkc2tTiUCYAJL1kuuO2724qlqlI+pbt17d4/+dQjGs8pVxXOvqQxBgRUbPDD9eJiKhLYpyEpCW9z2pO1+KB7sWUeKbXtXgGRdKlku6XdMcUxyXp05LWS1qtKdYJqktinFzX/ifqWjzQvZgSz/S6Fs+gfIHpJ1E+AVhQbkuAS/q5aBJjRAwt29cD03UCnkwx14DLYV27S+r59s+sGq4zvixB1641CF2LBwYT0+GHD+bV73333ZeRkZFtjuemm24aRDhA9/6bDSoe29s0mcnixYu9cePGvs696aab1gDVBc+W2V42g9vtDdxT2d9Qlt033ZdmVWKM4bNqVbfGY7c4gdEOY+PGjX3/d5f0G9sj2zmkrSQxRkTjGhwmeC+wT2V/flk2rbQxRkSjDIyOjfW1DcBy4A/L3ulXAw/bnvYxGlJjjIjGGTOYGqOkrwDHAPMkbaCYOvAZALb/HlgBnAisBx4D3tHPdZMYI6JZhrEBPUnbnnbOVBfP7O+d6XWTGCOicV1/FTmJMSIaZWAsiTEiYqLUGCMiKmwPqsd5u0lijIjGpcYYEVEzqOE620sSY0Q0quh8aTuK6SUxRkTj8igdEVE1BJ0vrb8rLelnkua1HUdENMMUNcZ+trakxhgRjev6AO9Ga4ySdpX0LUm3SbpD0qnlofdJulnS7eWSoePnXirpBkm3SDq5yVgjYvvpeo2x6UfpxcAvbB9i+0DgH8ryjbYXUazHcHZZ9ufA92wfAfw2cKGkXesXlLRE0ipJ3ZrxNCKm4L5/taXpxHg7cJykCyQdbfvhsvzq8vebgP3Kz8cD50i6FbgO2AXYt35B28tsj7Qxy29EzJzL2XX62drSaBuj7TvL5QtPBM6X9N3y0OPl76OVmAScYntdkzFGxPY3ll7pLSTtBTxm+0vAhcB0a7yupGh7VPndwxoIMSK2s/HZdfrZ2tJ0r/RBFG2FY8CTwHuAq6Y492PAfwNWS9oJ+ClwUiNRRsR2lQHeFbZXUtQEq/arHF9FMU05tn8NvLup2CKiIS3XBvuRcYwR0bjUGCMiKgyMJjFGREyUGmNERE0SY0REhdP5EhGxtdQYIyJqkhgjIiqKXuluvxKYxBgRjcuaLxERVS3PtdiPJMaIaNT40gZdlsQYEY3LcJ2IiJrUGCMiKjwEy6cmMUZE49pcz6UfSYwR0biuD9dpejGsiNjBjfdKD2r5VEmLJa2TtF7SOZMc31fSteUyzKslndjrmkmMEdG4QSVGSXOAi4ATgIXA6ZIW1k77C+BK24cBpwEX97puHqUjolmD7Xw5Alhv+y4ASVcAJwNrq3cEnld+3g34Ra+LJjFGRKMGPMB7b+Ceyv4G4MjaOR8FrpH0PmBX4NheF82jdEQ0bgbLp86TtKqyLXkatzsd+ILt+RRr2l9erjw6pdQYI6JxMxius9H2yDTH7wX2qezPL8uqzgQWA9j+gaRdgHnA/VNdNDXGiGic3d/WhxuBBZL2l/RMis6V5bVzfg68DkDSK4BdgH+d7qKpMUZEo8zg3pW2/ZSksyjWq58DXGp7jaTzgFW2lwMfBD4r6b+Ut3+7ezRyJjFGRLMG/Eqg7RXAilrZ0srntcBrZnLNJMaIaFSmHYuImEQSY0RETeZjjIiYwJ2fXafTw3Uk7SrpW5Juk3SHpFPbjikitk2/Q3XarFR2vca4GPiF7dcDSNqt5XgiYgC6PlFtp2uMwO3AcZIukHS07YfrJ0haMv66UAvxRcQMjY9j7POVwFZ0OjHavhNYRJEgz5e0dJJzltke6fHaUER0yCDnY9weOv0oLWkv4EHbX5L0b8A7244pIrZR1pXeZgcBF0oaA54E3tNyPBExCEmMT5/tlRTvQEbELDI2msQYEbFZMRQniTEiYoIkxoiICdL5EhGxFXd8YekkxohoVNoYIyIm4Y6/EpjEGBGN63iFMYkxIhpmp40xIqIubYwRERVZ8yUiYhJJjBERVTYeTa90RMQEqTFGp8yd+4K2Q5jgXx95pO0QogUdz4tJjBHRrHS+RETU5ZXAiIg6M5bOl4iIiVJjjIioyOw6ERGTSWKMiJjI3W5iTGKMiOblUToiospmLBPVRkRsMQwDvHdqO4CI2MG4WAyrn60fkhZLWidpvaRzpjjn9yStlbRG0v/sdc3UGCOieQOqMUqaA1wEHAdsAG6UtNz22so5C4BzgdfYfkjSC3tdNzXGiGhYsa50P1sfjgDW277L9hPAFcDJtXPeBVxk+yEA2/f3umgSY0Q0bmzMfW3APEmrKtuS2qX2Bu6p7G8oy6oOAA6Q9M+Sfihpca/4Gn+UlrQ7cIbti5u+d0S0z2UbY5822h7ZxlvuDCwAjgHmA9dLOsj2v031hTZqjLsDf9zCfSOiIwb4KH0vsE9lf35ZVrUBWG77Sds/Be6kSJRTaiMx/hXwMkm3Srqw3O6QdLukUwFU2Ko8ImaHASbGG4EFkvaX9EzgNGB57ZyvU9QWkTSP4tH6ruku2kav9DnAgbYPlXQK8J+BQ4B5FD1K1wP/ATi0Xm77vvrFyjaHertDRHRW30mv95XspySdBawE5gCX2l4j6Txgle3l5bHjJa0FRoEP2X5guuu2PVznKOArtkeBX0n6R+BV05TX/yXA9jJgGYCkbo8ajYiBT1RrewWwola2tPLZwAfKrS9tJ8aI2MEY8Gi36zBttDFuAuaWn78PnCppjqQ9gf8I3DBNeUTMAgNsY9wuGq8x2n6gHE90B/BtYDVwG8U/JB+2/UtJXwN+q17edKwRsR20nPT60cqjtO0zakUfqh13WfYhImLWmcE4xlakjTEiGpcaY0RExTBMO5bEGBHNsnEmqo2ImChrvkRE1ORROiKiKutKR0RMlM6XiIitmLHRbjcyJjFGRLPyKB0RMYkkxoiIiTqeF5MYI6JZ6XyJztm06cG2Q5jghbvt1nYIE3TxB1ZS2yEM1swWw2pFEmNENMyM5ZXAiIiJulgzr0pijIjmJTFGRGzhtDFGRGyt4xXGJMaIaFrWfImImMikVzoiosqkjTEiYit5lI6ImMCd731JYoyIZmXasYiIrY2NJjFGRGyW2XUiIuqG4FF6p+19A0k/kzRvG75/qKQTBxlTRLSpGODdz9aW7Z4Yt4WknYFDgSTGiFmk64lxoI/SknYFrgTmA3OAj5WH3ifpDcAzgLfY/omkFwCXAi8FHgOW2F4t6aPAy8rynwOvAZ4t6Sjg47a/OsiYI6J5XR/gPega42LgF7YPsX0g8A9l+Ubbi4BLgLPLsv8K3GL7YODPgC9WrrMQONb26cBS4Ku2D50sKUpaImmVpFUD/rNExHYwPrtOP1s/JC2WtE7SeknnTHPeKZIsaaTXNQedGG8HjpN0gaSjbT9cll9d/n4TsF/5+SjgcgDb3wP2kPS88thy27/u54a2l9kesd3zDxsR3TCoR2lJc4CLgBMoKlSnS1o4yXlzgfcDP+onvoEmRtt3AosoEuT5kpaWhx4vfx+lv8f3RwcZV0R0yUA7X44A1tu+y/YTwBXAyZOc9zHgAuA3/Vx0oIlR0l7AY7a/BFxIkSSn8n3greX3jqF43H5kkvM2AXMHGWdEtGiwj9J7A/dU9jeUZZtJWgTsY/tb/YY46Efpg4AbJN0KfAQ4f5pzPwocLmk18FfA26Y471pgoaRbJZ06yGAjoh0zqDHOG+9DKLclM7mPpJ2ATwAfnMn3BtorbXslsLJWvF/l+CrgmPLzg8CbJrnGR2v7DwKvGmScEdGeGb75srFH/8G9wD6V/fll2bi5wIHAdeUytC8Clkt6Y5mPJpU3XyKiYcaDm6j2RmCBpP0pEuJpwBmb71R0AG9+wUTSdcDZ0yVF6PgA74iYhQwe62/reSn7KeAsiifVHwNX2l4j6TxJb3y6IabGGBGNG+RbLbZXACtqZUunOPeYfq6ZxBgRjev6JBJJjBHRqEw7FhFRZzM2mlUCIyImSo0xImIik8QYEbGZh2AG7yTGiGiYcT+DFFuUxBgRjUuNMSKiZmxwrwRuF0mMERXlRAOd0qXa1cjIts8HXcyck8QYETFRh5L9ZJIYI6JxGa4TEVHTpeaBySQxRkTDzNjYaNtBTCuJMSIalQHeERGTSGKMiKhJYoyImMAZrhMRUWcywDsiYjM7rwRGRNQ4bYwREXV5VzoioiY1xoiImiTGiIgqZ7hORMQEBsacd6UjIirSK73dSVoCLGk7jojoXxLjdmZ7GbAMQFK3/7YjAkhijIiYoOh7yTjGiIgK446/ErhT2wH0S9IKSXu1HUdEbDv3+astQ1NjtH1i2zFExGCkjTEiYoKsKx0RMcEwrPkyNG2METF72O5r64ekxZLWSVov6ZxJjn9A0lpJqyV9V9JLel0ziTEiGjc2NtbX1oukOcBFwAnAQuB0SQtrp90CjNg+GLgK+Ote101ijIiGGTzW39bbEcB623fZfgK4Ajh5wt3sa20/Vu7+EJjf66JJjBHRuBkM15knaVVlq7/+uzdwT2V/Q1k2lTOBb/eKL50vEdGoGXa+bLQ9Moj7Svp9YAR4ba9zkxgjonED7JW+F9insj+/LJtA0rHAnwOvtf14r4smMUZEwwY6jvFGYIGk/SkS4mnAGdUTJB0GfAZYbPv+fi6axBgRjRvU8qm2n5J0FrASmANcanuNpPOAVbaXAxcCzwX+lySAn9t+43TXTWKMiEYNeoC37RXAilrZ0srnY2d6zSTGiGhY1nyJiNiKybvSERETdP1d6STGiGiYB9b5sr0kMUZEo7K0QUTEJPIoHRFRk8QYETFBhutERGylzYWu+pHEGBGNsmFsbLTtMKaVxBgRDet/2YK2JDFGROOSGCMiapIYIyJqMsA7IqLKGa4TETGBgbGO1xi3eZVASdeVi13fWm5XVY4tkfSTcrtB0lGVYydJukXSbeVi2O/e1lgiYjjYY31tbXlaNUZJzwSeYfvRsuittlfVzjkJeDdwlO2NkhYBX5d0BPAAsAw4wvYGSc8C9iu/93zbDz29P05EdF/3h+vMqMYo6RWS/hZYBxzQ4/Q/BT5keyOA7ZuB/wG8F5hLkZQfKI89bntd+b1TJd0h6YOS9pxJfBExHGz3tbWlZ2KUtKukd0j6J+CzwFrgYNu3VE77cuVR+sKy7JXATbXLrQJeaftBYDlwt6SvSHqrpJ0AbP89cALwHOB6SVdJWjx+PCKG2/iaL11OjP08St8HrAbeafsnU5yz1aN0L7bfKekg4FjgbOA44O3lsXuAj0k6nyJJXkqRVLda2UvSEmDJTO4dEW0y7vgrgf3Uwt5MsV7r1ZKWSnpJn9deCxxeKzscWDO+Y/t225+kSIqnVE8s2yIvBj4NXAmcO9lNbC+zPWJ7pM+4IqJl7vNXW3omRtvX2D4VOBp4GPiGpO9I2q/HV/8auEDSHgCSDqWoEV4s6bmSjqmceyhwd3ne8ZJWA+cD1wILbf+J7TVExKwwGx6lAbD9APAp4FNlba5aF/6ypF+XnzfaPtb2ckl7A/8iycAm4Pdt3ydpLvBhSZ8Bfg08SvkYTdEh8wbbd2/TnywiOqvrvdJPa7iO7Rsqn4+Z5rxLgEsmKd8EnDjFd+odNhExixS1wW4P8M6bLxHRuFlZY4yI2BZZPjUioi41xoiIKmNSY4yI2Gz8zZcuS2KMiMYlMUZE1CQxRkRM4CyfGhFRNQxtjJnKKyKaN77uS6+tD+W0hOskrZd0ziTHnyXpq+XxH/Uxz0MSY0Q0rd+5dXonRklzgIsopidcCJwuaWHttDOBh2z/O+CTwAW9rpvEGBGNG+CaL0cA623fZfsJ4Arg5No5J1OsHgBwFfA6SZruomljjIjGDfCVwL2Beyr7G4AjpzrH9lOSHgb2ADZOddHZlhg3Us7ruI3mMc1fWgu6Fg90L6ZZG0+Pyk2/BhVPvxNVT2clRTz92EVSdXWAZbaXDSCGac2qxGh7IItnSVrVpRnBuxYPdC+mxDO9LsVje/EAL3cvsE9lf35ZNtk5GyTtDOxGuRDfVNLGGBHD7EZggaT9y2WdT6NYaK9qOfC28vObge+5x3ihWVVjjIgdS9lmeBbF4/kc4FLbaySdB6yyvRz4PHC5pPXAgxTJc1pJjJPb7m0YM9S1eKB7MSWe6XUtnoGxvQJYUStbWvn8G+AtM7mmuj4CPSKiaWljjIioSWKMiKhJYoyIqElijIioSWKMiKhJYoyIqElijIio+f9HiZHWQwggIQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate_and_show_attention(\"elle est trop petit .\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input = je ne crains pas de mourir .\n",
      "output = i m not afraid of dying . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUYAAAEZCAYAAADrD4zSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHClJREFUeJzt3XmcXWWd5/HPl4CyusZxYW8NYkAEEgEVFBXtqAjSLoBoC6Jpp8VlELux24FuRmdeaquNIy4RAXVQW1E0QmwYFwYVERIQMJHYeYFKsG07gOyypL7zxzkFp24qdW+l7j3n3Fvfd17nVWe753nqJvx4zrPKNhER8bDNms5ARETbJDBGRHRIYIyI6JDAGBHRIYExIqJDAmNERIcExoiIDgmMEREdEhgjIjokMEbMMip8S9Izms5LWyUwRsw+LwWeDbyl6Yy0VQJjREVZmtqx6XwM2PEUQfGVkjZvOjNtlMAYUeFiVpVlTedjUCTNBfaw/V3ge8CrGs5SKyUwRmzoKknPbjoTA/JG4Cvl/tnkdXpSyrRjERNJuh54GvAb4G5AFIXJvRrNWB9Iug5YZPvm8vga4FDbNzWbs3ZJ/cIIkfRa4F9t3ynp/cC+wAdsX9Vw1obNnzedgUGQ9Bjgk+NBsXQSMBdIYKxIiXGESLrW9l6SDgQ+AHwEOMX2/g1nbShIepTtOyQ9brLrtm+tO0/RjNQxjpb15c9XAEtsXwg8osH8DJsvlz9XAMvLnysqx0NL0lslzSv3JelsSXdIulbSPk3nr21SYhwhki4AbgZeQvEafS9whe1nNZqxISJJwI62f9t0XvpJ0i+AfWw/IOn1wHso+jPuA5xq+6BGM9gyKTGOltcBFwF/bvuPwOOA9zabpeFSdte5sOl8DMCDth8o9w8Fvmj7FtvfA7ZpMF+tlMA4QmzfA3wbuFvSTsAWwPXN5moojWJ3nTFJT5a0JfBiij6M47ZqKE+tlVbpESLpHcCpwH8AY+VpA0PfzaRm+wPHSBql7jqnUNSTzgGW2l4JIOkFwA1NZqyNUsc4QiStAfa3fUvTeRlmknae7Lzt39Sdl34qh/9tZ/u2yrltKOLAXc3lrH1SYhwtNwG3N52JETCqpYXHAW+XtEd5vBL4lO3/aDBPrZTAOFpuAC6RdCFw3/hJ2x9rLktD6UKK4ChgS2BXYDWwx1QfajNJz6PojnQO8MXy9ALgZ5KOsf2TpvLWRgmMo+W35fYI0n9xk9l+ZvVY0r7AXzeUnX75KPAq21dXzi2VdD7wWYp61SiljjGiB5Ku6wyYw0TSKtvzp3tttkqJcQRI+mfb75b0HSapH7N9WA152AzY1vYdg05r0CSdWDncjKKz/O8ayk6/SNJjqw0v5cnHkW57G0hgHA1fKn/+U52JSvoy8DaKoYhXAo+SdLrtj9SZjwHYrrL/IEWd4zcayku/fBy4WNJJwPikIguAD5XXoiKv0rHJJP3c9t6SjqEoVZ0MrBjy/n4PkbQtwKh0ZZF0KPA3FI1IBlYBH7H9nUYz1kIpMY6QcpKA/wXMp2hNBcD2nw0oyS0kbUExC/Qny3G4Q/9/Wkl7UpTCH1cerwPeZPsXjWZshmxfAFzQdD6GQeoWRsvZwKcpXv9eSNEt4/8MML3PAr+mGGt7adkxeujrGIElwIm2d7a9M8WEC0saztOMSPpaZf9DHdcurj9H7ZZX6REiaYXtBdUW1PFzNeZhc9sP1pXeIEi6pnNGosnODRNJV9vep9y/yva+k12LQl6layBpN4qS3BNt7ylpL+Aw2x/oc1L3la3D/ybpBIopyLbtcxoTSHoFRZ3VlpXTpw0gnbq+Q4AbJP13Hm7UegPDP554qhJQSkcd8ipdj88B7wMeALB9LXDUANJ5F7A18E6KFsc3AG8aQDoASPoMcCTwDopRIq8FJh1n3Ad1fYcAbwaeQNES/Q2Kqf+PG1Baddla0j6SFgBblfv7jh83nbm2mZUlxnLq/3m2z5b0BIr+dzcOMMmtbV9RzIH6kL6+bkqaAxxp+yTgLur5D/m55VIK19r+R0kfBb47oLQG/h1WPBXYkaLgsDnFNF0vYrhnKfp3YHxo6O8r++PHUTHrAqOkU4GFwNMpGiu2oGigeN4Ak10n6amUryySXkPxD7VvbK8vA36d7i1/3iPpKcAtwJMHlNbAv8OKcykWifoFD0/fNtRsv7DpPAyTWRcYgSMopnO/CsD27yRtN/VHZuztFK2au0u6GbgROGYA6VwtaSnwdYp5BAGw/c0BpAVwQbny3Icp1kUBOHNAadX1HQL85yj27ZO0FbCb7Wsq53YC1nesHDjrzbpWaUlX2N5vvGWunI/up4PslCzpkcBrgF0o+sbdQTHxaV8bKSSdPclp235zP9OppLcV8F+BgyhKcj8CPm37T31M48SOU1tRvOLeDYOZOUjSi4Gjge8zcZaiQf0PphZln9Prgb1s312euxj4O9tDvdhXv83GEuPXJH0WeIyktwLHM7hSzrhvA3+kKKUOcsztZsC7yvVekPRYillVBuULwJ3AJ8rj11P0nXxdH9MYL80/HXg2xXcp4I3AFX1Mp+o4YHeKapbqTOhDHRjLDvjnU/z9nF2WFp+QoLihWVdiBJD0EooV0gAuKhcEGmR6v7C95yDTKNPZoD/aIPuoTTYry6BmapF0KfAK23eWx9sBF9p+/gDSWm376f1+bhtI2p1iad3nS3o/cIftT3T73Gwza7rrSPpx+fNOii4Ybyu38yXdLulGSYOac+8ySXVMWbVZWUoEHpo5ZZBvBVdJOqCS3v4Mbv3lJwL3V47vL88NwmWSRnIaLtvXU8y0sxtFd6cvdfnIrDRrXqVtH1j+nLShRdLjgcuATw0g+QOBYyXdSFFnNajFlT4K/FTS18vj1wIf7HMaVQsogsj4Gsw7AaslXUf/f78vAleUr4JQjM8+p4/PrzoA+HkNf18bJelJtgfVjebzFNVH13VOQxaFWfkqvTGSnmy7711A6lxcqSzpvKg8/IHtVf1Oo5LWlJ25+/37lTNpjy8Mf2nHbNT9TKfxxbAkXWj7FQN69tYUXZ1ePehqpGGVwBgR0WHW1DFGRPQqgRGQtDhpJa2m06o7vbp/t0GQdJakP0iadK5MFT4haY2ka8vqmK4SGAt1/gNJWkmrLekNfWCkaIBbNMX1lwHzym0xxQxNXSUwRsTQsn0pcOsUtxwOfNGFyykGdnQdzz9S3XVmMq1+nVPyTzetBQs2bZ7ZnXbaiYULF07791qxYkX3mybR5u9wWNKqO71NScu2ut+1cYsWLfK6det6unfFihUrgeoQ0yW2pzOb+vbATZXjteW5KXufjFRgHFXLl9c7Yqtjaq+Ivlq3bl3P/6Yl/cn2wgFnaQMJjBFRuxq7Cd5MMbfmuB3Kc1NKHWNE1MrA+rGxnrY+WAr8Zdk6fQBwey+DOFJijIiaGfdpmRlJXwEOBuZKWgucSjErErY/AywDXg6sAe6hx5ntExgjol6GsT69Sds+ust1U0xyPC0JjBFRu7YPRU5gjIhaGRhLYIyImCglxoiICtv9anEemATGiKhd20uMQ9OPUdJlTechIvrDPf5pytCUGG0/t+k8RMTMFY0vTediakMTGCXdZXvbpvMRETPX9lfpoQmMG1NOtjkK88pFzA5pfBm8cgqiJVD/9FARMX0mJcaIiA2kg3dERIeUGCMiJmi2K04vhiYwpkU6YjS4j7PrDMrQBMaIGB1jaZWOiHhYZteJiJhEGl8iIqrslBgjIjqlxBgRUWFgfQJjRMREKTHGjEmqNb06/9HW/btFOyQwRkRUOI0vEREbSokxIqJDAmNEREXRKp0hgRERE2QSiYiIKjuv0hERVVnaICJiEumuExHRISXGiIgKZ/nUiIgNZc2XiIgObe+us1nTGaiStIuk6yWdI+lXks6VdIikn0j6N0n7NZ3HiJiZ8VbpXrZeSFokabWkNZJOnuT6TpJ+KOlqSddKenm3Z7YqMJaeBnwU2L3cXg8cCJwE/F3nzZIWS1ouaXmtuYyITdavwChpDnAG8DJgPnC0pPkdt70f+JrtfYCjgE91e24bX6VvtH0dgKSVwPdtW9J1wC6dN9teAiwp7295AT0i6G/jy37AGts3AEj6KnA4sKqaIvCocv/RwO+6PbSNgfG+yv5Y5XiMduY3Iqahzx28twduqhyvBfbvuOcfgIslvQPYBjik20Pb+CodESNurJyTsdsGzB2vKiu3xZuQ3NHAObZ3AF4OfEnSlLEvJbCIqN00uuuss71wius3AztWjncoz1UdDywCsP1TSVsCc4E/bOyhrSox2v617T0rx8faPm+yaxExvOzeth5cCcyTtKukR1A0riztuOe3wIsBJD0D2BL4z6kemhJjRNTK9G+stO0HJZ0AXATMAc6yvVLSacBy20uB9wCfk/TfyuSPdZdKzgTGiKhXn4cE2l4GLOs4d0plfxXwvOk8M4ExImqVacciIiaRwBgR0SHzMUZETODMrhMRUTWNrjiNSWCMiNplotqIiIp+9mMclATGiKhdWqUjIqqyrnRExCQSGCMiJhpbn8AYEfGQortOAmNExAQJjBERE6TxJSJiA275wtIJjBFRq2GoY2zV0gYbI+lYSU9pOh8R0R8eG+tpa8pQBEbgWCCBMWJE9HHNl4FoJDBK2kXSLyV9TtJKSRdL2krS3pIul3StpPMlPVbSa4CFwLmSfi5pqybyHBF9YuOx3ramNFlinAecYXsP4I/Aq4EvAn9rey/gOuDUcpXA5cAxtve2fW/1IZIWj685W3P+I2ITuRwW2G1rSpONLzfa/nm5vwJ4KvAY2/+vPPcF4OvdHmJ7CbAEQFK7a3QjImu+dHFfZX898JimMhIR9Wp7YGxT48vtwG2SDiqP3wiMlx7vBLZrJFcR0V82Xj/W09aUtvVjfBPwGUlbAzcAx5XnzynP3ws8p7OeMSKGS9tLjI0ERtu/BvasHP9T5fIBk9z/DeAbg89ZRNSh5XGxdSXGiBhxaXyJiOg0BEMCExgjomZmrMGGlV4kMEZE7VJijIioGIbZdRIYI6J+CYwRERO53VWMCYwRUb+8SsfQkdR0Fgaizv8YR/U77AubsQYnoe1FAmNE1GoYOni3aRKJiJgNTF8nqpW0SNJqSWsknbyRe14naVU5MfaXuz0zJcaIqF+fSoyS5gBnAC8B1gJXSlpqe1XlnnnA+4Dn2b5N0n/p9tyUGCOiZr3N3t3j6/Z+wBrbN9i+H/gqcHjHPW+lWC3gNgDbf+j20ATGiKjd2Jh72oC540uXlNvijkdtD9xUOV5bnqvaDdhN0k/KNaUWdctfXqUjolYu6xh7tM72whkmuTnFGlMHAzsAl0p6pu0/buwDKTFGRO36+Cp9M7Bj5XiH8lzVWmCp7Qds3wj8iiJQblQCY0TUro+B8UpgnqRdJT0COApY2nHPtyhKi0iaS/FqfcNUD82rdETUrH9Lo9p+UNIJwEXAHOAs2yslnQYst720vPZSSasoFt57r+1bpnruQAKjpNcCpwG/t/3CaXzuMtvPneT8OcAF5RrTETHM+jy7ju1lwLKOc6dU9g2cWG49GVSJ8XjgrbZ/XD0paXPbD27sQ5MFxYgYLQa8vt0jX2YcGCV9i6Lyc0vgdOBJwIHA5yUtBVYCfwFsC8yR9Arg28BjgS2A99v+dvmsu2xvq2Kg6f+m6LR5E3D/TPMZEe3R9iGB/Sgxvtn2rZK2oqgIfQHwIuAk28slHQvsC+xV3rc5cITtO8qK0MvLnurVb+oI4OnAfOCJwCrgrMkSL/s1dfZtioi26r1hpTH9CIzvlHREub8jkzeD/1/bt5b7Av6npOcDYxSdMZ8I/L5y//OBr9heD/xO0g82lrjtJcASAEnt/rYjAphWP8ZGzCgwSjoYOAR4ju17JF1C8Urd6e7K/jHAE4AFth+Q9OuNfCYiRlTbS4wz7cf4aOC2MijuDhzQ42f+UAbFFwI7T3LPpcCRkuZIejLQc8t2RLTb+LRjferHOBAzfZX+V+Btkn4JrAYu7+Ez5wLfkXQdsBy4fpJ7zqeop1wF/Bb46QzzGRFtYeNRnqjW9n3Ayya5dHDlnnOAcyrH64DnbOR525Y/DZwwk7xFRHtlzZeIiA5tr2NMYIyIemVd6YiIiYZhzZcExoiomRlb3+5KxgTGiKhXXqUjIiaRwBgRMVHL42ICY0TUK40vES1SzGY3muoKNAsXznRdKoo6xlGeRCIiYvrM2CgPCYyI2BR5lY6I6JTAGBHxMKeOMSJiQy0vMCYwRkTdZseaLxERvTNplY6IqDKpY4yI2EDbX6VnuhhWLSS9U9IvJZ3bdF4iYqZcNk33sDVkWEqMfw0cYntt0xmJiBnKtGPTJ+lE4M3l4ZnA7sCfAd+VdJbtjzeWuYjoi7H1CYw9k7QAOA7YHxDwM+ANwCLgheUKgxExxDK7zvQdCJxv+24ASd8EDprqA5IWA4tryFtE9ENepQfP9hJgCYCkdn/bEcEwdPBuW6v0j4BXSdpa0jbAEeW5iBghtnvamtKqEqPtqySdA1xRnjrT9tWjPMFoxGzU9g7ebSsxYvtjtvcst38uz+2ShpeI0TA+u04vWy8kLZK0WtIaSSdPcd+rJVlS12nIWxcYI2L09etVWtIc4AzgZcB84GhJ8ye5bzvgXRQ9XbpKYIyImvUWFHusY9wPWGP7Btv3A18FDp/kvv8BfAj4Uy8PTWCMiHr191V6e+CmyvHa8txDJO0L7Gj7wl6z2KrGl4iYHabR4jxX0vLK8ZKyi15PJG0GfAw4tvfcJTBGRM2mOfJlne2pGktuBnasHO9Qnhu3HbAncEnZu+VJwFJJh9muBtwJEhgjombG/Zuo9kpgnqRdKQLiUcDrH0rJvh2YO34s6RLgpKmCIqSOMSLqZvBYb1vXR9kPAicAFwG/BL5me6Wk0yQdtqlZTIkxImrXz1EttpcByzrOnbKRew/u5ZkJjBFRu7aPlU5gjIhaZdqxiIhONmPrs0pgRMREKTFGRExkEhgjIh7izOAdEdHJuJdOig1KYIyI2qXEGBHRYax/QwIHIoExImpVzLXY7sA4kLHSkv5B0klTXD9zsll2I2KWKFpgum8NaaTEaPstTaQbEe3Q9u46fSsxSvp7Sb+S9GPg6cAcSVdVrs8bP5Z0yfiCNJLukvRBSddIulzSE8vzTy2Pr5P0AUl39SuvEdGsti+f2pfAKGkBxTxoewMvB54NrAdul7R3edtxwNmTfHwb4HLbzwIuBd5anj8dON32MymmK4+IkWDGxtb3tDWlXyXGg4Dzbd9j+w5gaXn+TOC4ciWvI4EvT/LZ+4ELyv0VwC7l/nOAr5f7k30OAEmLJS3vmP48IlpqvIP3yJcYp/ANimUNDwVW2L5lknse8MPfwHqmWe9pe4nthV2mP4+IFpktgfFS4FWStirXb30lgO0/Ucys+2kmf42eyuXAq8v9o/qUz4hogVkRGG1fBfwLcA3wXYp1GMadC4wBF0/zse8GTpR0LfA04PY+ZDUiGtdjV51R6K5j+4PABye5dCBwtu31lXsPruxvW9k/DzivPLwZOMC2JR1F0dIdESPAtLuD90D7MUo6H3gq8KJN+PgC4JMq1jz8I/DmfuYtIpphz/IhgbaPmMFnfwQ8q4/ZiYhWaLb+sBcZKx0RtWv7WOkExoioXUqMEREdEhgjIqoa7orTiwTGiKiVgTE3Nw66FwmMEVGztEpHRGwggTEiokMCY0RERdH2kn6MEREVxrN5SGBExGTavuZLAmNE1C51jBERE7R/XekExoio1fiaL2026DVfIiI20M+lDSQtkrRa0hpJJ09y/URJqyRdK+n7knbu9swExoio3djYWE9bN+UKpGdQLLo3Hzha0vyO264GFtrei2KFgA93e24CY0TUzOCx3rbu9gPW2L7B9v3AV4HDJ6Rm/9D2PeXh5cAO3R6awBgRtXOPf4C54+vGl9vijkdtD9xUOV5bntuY4ykW7JtSGl8iolbTbHxZ16814yW9AVgIvKDbvQmMEVG7PrZK3wzsWDneoTw3gaRDgL8HXmD7vm4PHfrAWBatO4vXEdFafe3HeCUwT9KuFAHxKOD11Rsk7QN8Flhk+w+9PHToA6PtJcASAEnt7hwVEUD/lk+1/aCkE4CLgDnAWbZXSjoNWG57KfARYFvg68VqzPzW9mFTPXfoA2NEDJd+d/C2vQxY1nHulMr+IdN9ZgJjRNSs/Wu+DE13HUnLJD2l6XxExMyZsZ62pgxNidH2y5vOQ0T0R9vHSg9NYIyIUeG+Nb4MSgJjRNQqSxtEREwir9IRER0SGCMiJmh/d50ExoioXRbDioiosGFsbH3T2ZhSAmNE1Kz3ZQuaksA4BOr+R1QOtI8hMmx/ZwmMEREdEhgjIjqkg3dERJXTXSciYgIDYykxRkRMlFfpiIgJ0l0nImIDCYwRERX9XvNlEBIYI6Jmxi0fEjjjNV8kXSJptaSfl9t5lWuLJV1fbldIOrBy7VBJV0u6RtIqSX8107xExHBwj3+askklRkmPALawfXd56hjbyzvuORT4K+BA2+sk7Qt8S9J+wC0Ua0HvZ3utpEcCu5Sfe6zt2zbt14mIYdD2V+lplRglPUPSR4HVwG5dbv9b4L221wHYvgr4AvB2YDuKoHxLee0+26vLzx0p6ReS3iPpCdPJX0QMB9s9bU3pGhglbSPpOEk/Bj4HrAL2sn115bZzK6/SHynP7QGs6HjccmAP27cCS4HfSPqKpGMkbQZg+zPAy4CtgUslnSdp0fj1SfK3WNJyScsnux4R7VIEvbGetqaoW1SWdAdwLfAW29dPcv0S4KRJXqVvBXa1fXvl3OHAm2z/RXn8TOAQ4C+Ba2wf2/EMUQTJM4Hltg/rktd2l883UWbXiTaxPaN/IHPmbO5ttnl0T/feeeetK2wvnEl6m6KXV+nXADcD35R0iqSde3z2KmBBx7kFwMrxA9vX2f448BLg1dUby7rITwGfAL4GvK/HdCOi5cbGxnramtI1MNq+2PaRwEHA7cC3JX1P0i5dPvph4EOSHg8gaW/gWOBTkraVdHDl3r2B35T3vVTStcAHgB8C822/2/ZKImI0jE8k0W1rSM+t0rZvAU4HTi9Lc9WOSOdKurfcX2f7ENtLJW0PXFa+4t4JvMH2v0vaDvgbSZ8F7gXupgiaUDTIvNL2b2b0m0VESxnT7rHSXesYh0nqGPsjdYwxlZnWMW622RxvueXWPd177713NVLHmJEvEVG7thfIEhgjonYJjBEREzjLp0ZEVA3D7DoznkQiImLa+thdpxwZt1rSGkknT3L9kZL+pbz+sx66GiYwRkTdep1bp3tglDQHOINihNx84GhJ8ztuOx64zfbTgI8DH+r23ATGiKhdH8dK7wessX2D7fuBrwKHd9xzOMUENgDnAS9Wlz5pqWOMiNr1cbjf9sBNleO1wP4bu8f2g5JuBx4PrNvYQ0ctMK6jHFo4TXOZ4kvqs2mnNYMO163+vZJW4+ltSlq9zpUwlYvKtHuxZcfMWUtsL+lDHqY0UoHR9ibN3yhpeV2965NW0mpLenX/buNsL+rj424Gdqwc71Cem+yetZI2Bx5NORfsxqSOMSKG2ZXAPEm7lisLHEUx12vVUuBN5f5rgB+4S3+hkSoxRsTsUtYZnkDxej4HOMv2SkmnUczhuhT4PPAlSWuAWymC55QSGAsDr7NIWkmrhenV/bsNhO1lwLKOc6dU9v8EvHY6zxyp2XUiIvohdYwRER0SGCMiOiQwRkR0SGCMiOiQwBgR0SGBMSKiQwJjRESH/w+i+BEFlp/KiQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate_and_show_attention(\"je ne crains pas de mourir .\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input = c est un jeune directeur plein de talent .\n",
      "output = he s a talented talented person . <EOS>\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEgCAYAAABYaaN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xu8XGV97/HPN/ECSCC2REUghEODCIhAUiwVKahQUAq10ALCOYeLxgtYWysV2744FbVekKp4UAkV8IL1gtbmKDURimJVNDvcQgJpKaiAthhuBlQg2d/zx1rbTIbZe/ZO1sya2ev75rVemXnmmfU8s0l+8+znKttEREQzzKi7AhER0T8J+hERDZKgHxHRIAn6ERENkqAfEdEgCfoREQ2SoB8R0SAJ+hERDZKgH0NF0kxJf153PSKGVYJ+DBXbG4CT6q5HxLBStmGIYSPpg8BTgc8Dj46l276htkpFDIkE/Rg6kq7tkGzbL+17ZSKGTIJ+RESDPKXuCkRMlaRzO6XbPq/fdYkYNgn6MYwebXm8FXA0cFtNdYkYKuneiaEn6enAUtuH1l2XiEGXKZsxHWwD7Fx3JSKGQbp3YuhIWgmM/Yo6E5gDpD8/YhLSvRNDR9KuLU/XA/9te31d9YnBIUnAPwFvt51xng7SvRNDx/aPgF2Al9q+F5gtabeaqxWD4Qjgt4HX1F2RQZWgH0NH0v8B3ga8vUx6GvCZ+moUA+QMioD/B5LSfd1Bgn4Mo1cBx1BO3bT9E2BWrTWK2knaAdjb9r8AVwN/WHOVBlKCfgyjx10MRhlA0jNqrk8Mhv8J/GP5+DLSxdNRgn4Moy9IupiiL/+1FK26f6i5TlG/0ymCPbaXAztK2qXeKg2ezN6JoSTpcIpBO1EszPpGzVUaGpKebvuxbmnDRNJs4ATbF7ekHQ6stX1jfTUbPAn6UZlyKuV821dL2hp4iu11PSjnfbbf1i0tOpN0g+0DuqXF9JTunahE2c1yJTDW0toZ+EqPiju8Q9pRPSpr2pD0HEkLgK0l7S/pgPI6lGJV81CS9FpJ88vHknSZpJ9LukXS/nXXb9BkSlNU5UzgQOD7ALb/Q9KzqixA0huANwK7S7ql5aVZwHerLGua+n3gVIov5L9vSV8H/FUdFarIm4HLy8cnAfsCuwH7AxcCL6mnWoMpQT+q8pjtx4sFkVDOka667/CzwL8A7wHOaUlfZ/uBisuadmx/EvikpONsf6nu+lRove0nysdHA5+yfT9wtaT311ivgZSgH1X5lqS/oug6OJyiRf7/qizA9sPAw5I+DDwwNl4gaTtJL7L9/SrLm8a+KunVwDxaYsAQn0cwKmlH4EHgZcC7W17bup4qDa4E/ajKORSrIVcCrwOuonfTKD8GtA46PtIhrRKS5gCv5ckB8vSqy+qjfwYeBlYAQztjp8W5wAjF5ntLbK8CkPR7wJ11VmwQZfZODB1JN9nery3tFtv79qCs7wLfpgiQG8bSh7l7RNKttvepux5VKrsTZ9l+sCXtGRQx7pH6ajZ40tKPSkh6MfC3wK4Uf69EcVj5/+hBcXdK+lOK1j0UXUm9atFtMw2ngn5X0gtsr6y7IhX6DeBMSXuXz1cBH7X93zXWaSClpR+VkHQ78Oc8uUV8fw/KehbFrIyXUgwWXwP8me37elDWu4Dv2r6q6nvXRdJq4LeAuyi6d8a+oCv/TakfygbHZylm8KwokxcA/xs42fZ3aqraQErQj0pI+r7tF9Vdj6pJWgc8gyI4PsHGALldrRXbAm3nEfxauWX10JF0PfCG9pW3kvYDLp6Ofy+3RBZnRVWulXS+pINaFv30ZIWnpD0kXSPp1vL5vpL+phdl2Z5le4btrW1vVz4f2oAPTzqP4EfALxjuWLBdp60WbN9Edl99krT0oxKSru2QbNsv7UFZ3wLOpmjF7V+mVTo4KWlP27eP98Vl+4aqyuq38jyChcDzbO8h6bnAF22/uOaqbRZJtwG/2zqIW6b/BkXX3J711GwwZSA3KmH7sD4Wt43tH4wtBCtVfVziW4BFwAUdXjPFeMKwehXFatUboDiPQNIwt4g/CCyT9FbKz0TRp/++8rVokaBfg2m6y+G5ndJ7tOBnraTd2bif/vHAT6sswPai8s9+fpn1y+O2LWlanEdge7GknwDvBPam+HuxGniX7UoXCE4HCfr1+B5PXkjUKW2YPNryeCuK5fC9Opj6TGAxsKekeylmoZzci4IkbUPR6p9re1G5sdfzbH+1F+X1Sft5BKcDl9Rcpy1S/v8Y5v8nfZOg30eSngPsRLnLIcVMEIDtGOJdDgFsb9INIukDwNKqy5E0A1ho++VlC3VGL7ZvbnEZxTTA3y2f3wt8kSEOMLY/UG6V8XPgecC5w3wegaQv2P6T8vEmW2xLWmb7iPpqN3gS9PurdZfDC9gY9Id9l8NOtqH4nJWyPSrpL4Ev2H606xu23O62T5B0Uln+L9Q2mDCMyiA/tIG+zfyWx4cDrYvp5vS5LgMvQb+P6tjlsF8rZSWtZOOumjMp/rH1agOvq8tBu8/T0q3Uo502Hy8PhBnr/96dHu1XI2kPilXGz7a9j6R9gWNsv6ui+6+j886nw772YKIpiJme2CZBvx47S9qOooV/CUVf/jm2l/WgrE/QYaVsDxzd8ng98N+2q55RM+aE8s8zW9IM9GLLh/8DfB3YRdIVwIspflvrhUsop6IC2L5F0meBSoK+7WGeoTORbcru0hls2nUqssvmk2Sefg0k3Wz7hZJ+H3g98DfAp3txXF0/V8pKOpjiuMTLJO1AsQHWXf0ou5ck/SbwOxRB5Hrba3tUznLbvy3pxpb1B0/aXK7C8p5FMegOgO0f96KcXhtnjcivTdMZWJstLf16jPUJv5LiwIdVPewnvlbS+cCXaemWqHpxUeuCH4rBz6cBn6FoGVdVxktt/6ukP+r0uu0vV1hW+xfw2JTQuZLm9mhxVs+nopb3PYZiTOm5wH0UXX+3UUx3HDoJ6lOToF+PFZKWUnRHnFMujBntUVljrfwF5Z+iN4uL+rHg5xDgX4E/oPgMavuzsqDPpouyWn8d7tXPD/o3FfWdFL+5XG17f0mHAaf0oJy+Kcdd9rB9c0vaXGCD7Xvrq9ngSdCvxxkUXTqry9kgc4E/61FZ3+yQ1os+vX4s+Fkn6S3ArWwM9tCDzzPWeiyDyRuBg8tyvs3GLZ0rUX6mMVcB11L0Tz8KHMem59lW4Qnb90uaIWmG7WslfajiMvptPfBlSfu2zOr6B4pZcQn6LYZ5k6VKSfqkpNktz58p6dIeFXcR8GzgyPL5Oqr/hz3mkZZrfVnmvB6U077g5xqqPzlrW4oNtBYAbwB2pOiieD29W9j2SeD5FFs5fwTYC/hUxWXMKq+FFJ/rmcBseve5HpK0LXAdcIWK4yeH+qCR8ozcfwLG5uvPBebYHqm1YgMoA7ml1sGzidIqKusG2we0DdjdbPuFVZfVoeynA0ttH9qDex8OjC2EWWr76qrLKMu5DnilN56ROwv4mu1DelDWatt7dUurqKy+fC5JF1DMEppB0X20PfBC22dUWU6/SdoTWGz7kHLX1Z/bvrDueg2adO9sNEPSM8d26it36OvVz+cJSTPZOGA3h9716berdNGUpH+zfXDLHPCxLpfXSxoFHgDOt/3Rqsqk+C3p8Zbnj5dpvXCDpN+xfT2ApBdRnMfaC/36XIfZHqX4O/dJKI6b7EE5fVXuiqpyvcOJwEvqrtMgStDf6ALge5K+WD7/Y+DdPSrrQopfRZ8l6d3A8RR9/JXr9aIp2weXf3YctC2nO34XqDLofwr4gaR/Kp//IcWpSb2wgOJ4wbHpjHOBNWM/14pPm+rp55L0Borxid3bgvwsoK+nS0l6ju3/6sGtP0HRrbiyfavlKKR7p4Wkvdg4K+Nfba/uYVl7Ai+jaBlfY7snm5Np01OSer1oarw67Gi70qmH5ZTKsZbcdZ0O0aionI6nTI1xxadN9fJzSdqeYrzgPcA5LS+t69Fq5onq8jXbr+zBfbehmOZ6XK+6F4ddgn5ExIAqJ5McDdzX6ZCgcn3Ph4FXUJyAdmq3NSSZvRMRMbguZ+Msv06Oothwbj7FoT9dpxMn6HcgaVHKGo6ypuNnSlnDU06v2b6OYjLEeI6lWNXvcrLBbEk7TnTPdO90IGnE9sKUNfhlTcfPlLKGp5xOjjzySK9d2317phUrVqwCftWStNj24vZ8kuYBXx2ne+erwHtt/1v5/BrgbROtT8jsnYiICq1du5aRke6zeiX9qo4vpmkf9Me2BejX+6ZbWQsWLOieqYO5c+eycOHCKZW1YsWKzSprkH9+Kau+sjaznLW2t/jglT72oNwL7NLyfGe6bDsx7YN+bJnJtFiq0ruNRiMmbYun4BrYMNqvtZYsAc6S9DmKzRUf7jY9OkE/IqJSxhXtASjpH4FDgR0k3UNxqM9TAWx/nGKDvlcAd1BM2Tyt2z0T9CMiqmQYrah3x/ZJXV43m54g11WCfkRExQZ5VmSCfkREhQyMJuhHRDRHWvoREQ1hu5+zd6YsQT8iomJp6UdENEhVUzZ7YSA3XJM0T9KtddcjImKqioHc7ldd0tKPiKjYIHfvDGRLvzRT0iWSVklaJmlrSbtL+rqkFZK+XZ4+FRExOMqB3G5XXQY56M8HLrK9N/AQcBywGHiT7QXAWxnn3FVJiySNSOrfxjERERTdO7a7XnUZ5O6du2zfVD5eAcwDfhf4YsvGXE/v9MZyT+rF0N/dAyMiIIuzNtdjLY83AM8GHrK9X031iYiYlPTpV+PnwF2S/hiKA4ElvbDmOkVEtPGk/qvLMAV9gJOBMyTdDKyiOB8yImJgeBLTNTNls43tHwL7tDz/QMvLE50MHxFRu9FswxAR0QzZZTMiomEGeSA3QT8iokp2WvoREU2Sln5EREMY2JCgHxHRHGnpR0Q0SIJ+RERDOAO5ERHNkpZ+RESDJOhHRDREMXsn2zBERDRGnRuqdZOgHxFRpZpPxuomQT8iokJjxyUOqgT9iIiKZcpmRESDpKUfEdEQttmQQ1QiIpqjzjNwu0nQj4ioWKZsRkQ0xKDP3plRdwWmStIzJH1N0s2SbpV0Qt11ioho5XKu/kTXZEg6UtIaSXdIOqfD63MlXSvpRkm3SHpFt3sOY0v/SOAntl8JIGn79gySFgGL+l2xiAgqGsiVNBO4CDgcuAdYLmmJ7dUt2f4G+ILtj0naC7gKmDfRfYeupQ+sBA6X9D5JL7H9cHsG24ttL7S9sIb6RUSDjXXvVNDSPxC4w/adth8HPgcc26G47crH2wM/6XbToQv6tv8dOIAi+L9L0rk1VykiYhOj5Z76E13ADpJGWq723omdgLtbnt9TprX6W+AUSfdQtPLf1K1uQ9e9I+m5wAO2PyPpIeA1ddcpIqLVJKdsrq2gN+Ik4HLbF0g6CPi0pH3s8bf5HLqgD7wAOF/SKPAE8Iaa6xMRsYmKJu/cC+zS8nznMq3VGRTjnNj+nqStgB2A+8a76dAFfdtLgaV11yMiohNT2d47y4H5knajCPYnAq9uy/Nj4GXA5ZKeD2wF/Gyimw5d0I+IGGgVzd6xvV7SWRSN3JnApbZXSToPGLG9BPgL4BJJf07xfXOqu4wSJ+hHRFSoysVZtq+iGKBtTTu35fFq4MVTuWeCfkRExQZ5RW6CfkRExbKffkREYzi7bEZENIVd2ZTNnkjQj4ioWA5RiYhoiArn6fdEgn5ERMUyeycioimmsF9+HRL0IyKqlqAfEdEcoxsS9CMiGqGYspmgHxHRGAn6ERGNkYHciIhG8WiCfkREI6RPPyKiYZxtGCIimmOAG/oJ+hERlbIHuk9/Rt0VmCpJX5G0QtIqSYvqrk9ERDuXWzFMdNVlGFv6p9t+QNLWwHJJX7J9f2uG8ssgXwgR0XdVnpHbC8MY9P9U0qvKx7sA84FNgr7txcBiAEmD+9OPiGkpQb8ikg4FXg4cZPsXkr4JbFVrpSIiWtl4Q2bvVGV74MEy4O8J/E7dFYqIaDfILf1hG8j9OvAUSbcB7wWur7k+ERFPMnZO7kRXXYaqpW/7MeCouusRETGeDORGRDRJtmGIiGgSM5qB3IiI5khLPyKiIbLLZkRE0yToR0Q0hwe3Sz9BPyKiauneiaElqe4qxBT0K9jk78UEbEZziEpERDMM+uKsYduGISJisLk4GL3bNRmSjpS0RtIdks4ZJ8+fSFpdnjHy2W73TEs/IqJqFbT0Jc0ELgIOB+6hOD9kie3VLXnmA28HXmz7QUnP6nbftPQjIirV/dSsSXb/HAjcYftO248DnwOObcvzWuAi2w8C2L6v200T9CMiKjY66q4XsIOkkZar/bS/nYC7W57fU6a12gPYQ9J3JF0v6chudUv3TkREhVz26U/CWtsLt7C4p1CcHngosDNwnaQX2H5ovDekpR8RUbGKunfupTgSdszOZVqre4Altp+wfRfw7xRfAuNK0I+IqFhFQX85MF/SbpKeBpwILGnL8xWKVj6SdqDo7rlzopumeyciolKTDuoT38VeL+ksYCkwE7jU9ipJ5wEjtpeUrx0haTWwATjb9v0T3XfCoC9pNvBq2x/tku8R29tO4fO0vvdUYJntn0zhPfOAr9reZ3PKjIjomQp32bR9FXBVW9q5LY8NvKW8JqVb985s4I1TqOPmOBV4bo/LiIjoCwPe4K5XXboF/fcCu0u6SdIHJV0j6QZJKyW1zxcFQNLZkpZLukXSO8q0eZJuk3RJuWpsmaStJR0PLASuKMvYWtICSd+StELSUkk7lvdYIOlmSTcDZ1b4M4iIqFRFffo90S3onwP8p+39gLOBV9k+ADgMuEBtuy5JOoJi5PhAYD9ggaRDypfnUywi2Bt4CDjO9pXACHByWcZ64CPA8bYXAJcC7y7ffxnwJtsv3KJPHBHRS5MI+HUG/akM5Ar4uzKIj1IsEng28F8teY4orxvL59tSBPsfA3fZvqlMXwHM61DG84B9gG+U3yczgZ+WYwuzbV9X5vs0cNS4FS0WObQvdIiI6IvJ7q1Th6kE/ZOBOcAC209I+iGwVVseAe+xffEmicXA62MtSRuArTuUIWCV7YPa3j97CvXE9mJgcfnewf3pR8S0NMy7bK4DZpWPtwfuKwP+YcCuHfIvBU6XtC2ApJ0msQFQaxlrgDmSDirf/1RJe5eryx6SdHCZ7+Qu94yIqMXY1spD2b1j+/5yT4dbKRYK7ClpJUU//O0d8i+T9Hzge2X3zCPAKRQt+/FcDnxc0i+Bg4DjgQslbV/W70PAKuA04NKy5b5sSp8yIqJfbDzAh6hokH8NqUK6d6JJcnLWFluxpfvh7Ljzrj7trL/umu89b3/dFpe1ObIiNyKiYoPcmE7Qj4ioUoUrcnshQT8iokKDfkZugn5ERKXM6IbBHchN0I+IqFK6dyIiGiZBPyKiOQY45ifoR0wn/Zo//7/OOLd7porstu9ufSvrHW8+bYvvkYHciIgmmfzB6LVI0I+IqJQZHeBtGBL0IyIqlu6diIgmSdCPiGgGp08/IqJZBrihn6AfEVGteg9J6SZBPyKiSiazdyIimsKkTz8iolEGuXtnwoPRJc2W9MZuN5H0yOZWQNKpkp47xffMK8/tjYgYMC6n8HS5ajJh0AdmA12D/hY6FZhS0I+IGFjl1srdrrp0C/rvBXaXdJOkD0q6RtINklZKOrbTGySdLWm5pFskvaNMmyfpNkmXSFolaZmkrSUdDywErijL2FrSAknfkrRC0lJJO5b3WCDpZkk3A2dW+DOIiKjU6AZ3verSLeifA/yn7f2As4FX2T4AOAy4QG1b+kk6ApgPHAjsByyQdEj58nzgItt7Aw8Bx9m+EhgBTi7LWA98BDje9gLgUuDd5fsvA95k+4XdPpSkRZJGJI10yxsRUaWxXTYHtaU/lYFcAX9XBvFRYCfg2cB/teQ5orxuLJ9vSxHsfwzcZfumMn0FMK9DGc8D9gG+UX6fzAR+Kmk2MNv2dWW+TwNHjVdR24uBxQCSBndEJSKmn2l0ctbJwBxgge0nJP0Q2Kotj4D32L54k0RpHvBYS9IGYOsOZQhYZfugtvfPnkI9IyJqNNiLs7p176wDZpWPtwfuKwP+YcCuHfIvBU6XtC2ApJ0kPWsKZawB5kg6qHz/UyXtbfsh4CFJB5f5Tu5yz4iI2gxt947t+yV9p5weuRzYU9JKin742zvkXybp+cD3yu6ZR4BTKFr247kc+LikXwIHAccDF0ravqzfh4BVwGnApWV3zbIpfcqIiD4a6sVZtl89iTzbtjz+MPDhDtn2acnzgZbHXwK+1JLvJuAQ2theAbQO4v5lt3pFRPTboO+y2a17JyIipqiq7h1JR0paI+kOSedMkO84SZa0sNs9E/QjIirVPeBPJuhLmglcRDFTcS/gJEl7dcg3C3gz8P3J1C5BPyKiSmX3TrdrEg4E7rB9p+3Hgc8BnRbFvhN4H/Crydw0QT8iomKTbOnvMLaItLwWtd1mJ+Duluf3lGm/JukAYBfbX5ts3bLLZkREhcZW5E7CWttd++DHI2kG8PcU+5dNWoJ+RESljKs5ROVeYJeW5zuXaWNmUcyK/GY5Rf45wBJJx9gedwuaBP2IiCoZXM3BWcuB+ZJ2owj2JwK/nkJv+2Fgh7Hnkr4JvHWigA8J+hGxGT71ifP6VlY/V6++482nVXKfKupse72ksyh2OpgJXGp7laTzgBHbSzbnvgn6EREVq+qLyvZVwFVtaeeOk/fQydwzQT8iokJTGMitRYJ+RESVbEY3VNOp3wsJ+hERVUtLPyKiOUyCfkREI3ganZwVERFdGVc0Ub8XEvQjIiqWln5ERIOMVrMNQ08k6EdEVKjYRTNBPyKiOdK905mkp9heX2cdIiKqNshTNrf4EBVJ8yTdLukKSbdJulLSNpIWSPqWpBWSlkrascz/TUkfkjQCvFnSH0u6VdLNkq4r82wl6TJJKyXdKOmwMv1USV+W9HVJ/yHp/Vta/4iIqlV1Rm4vVNXSfx5whu3vSLoUOBN4FXCs7Z9JOgF4N3B6mf9pY4cHSFoJ/L7teyXNLl8/E7DtF0jaE1gmaY/ytf2A/YHHgDWSPmK79XQZyhNo2k+hiYjoAzM6uqHuSoyrqqB/t+3vlI8/A/wVxeb+3yg3958J/LQl/+dbHn8HuFzSF4Avl2kHAx8BsH27pB8BY0H/mnIfaSStBnZl0yPFsL0YWFzmGdzfsyJi2mnK4qz2T7gOWGX7oHHyP/rrN9qvl/Qi4JXACkkLupT1WMvjDWQwOiIGzCAH/aoORp8raSzAvxq4HpgzlibpqZL27vRGSbvb/n65R/TPKI4H+zZwcvn6HsBcYE1FdY2I6Kkm9OmvAc4s+/NXU3TNLAUulLR9Wc6HgFUd3nu+pPmAgGuAm4HbgY+V/f3rgVNtP1Z2FUVEDDA3YsrmetuntKXdBBzSnrH9dBfbf9Thfr8CnnRume3Lgctbnh899apGRPSWyeKsiIhGsKf5Ngy2f0gxUyciIqi3z76btPQjIiqWvXciIhokLf2IiAZJ0I+IaAo3Y8pmRERQbE8w6um/987AWrBgASMjI30pK4vHIqo3fP+uMnsnIqJREvQjIhokQT8ioiGKcdzM04+IaAjj6bwNQ0REbGqQz8hN0I+IqFj69CMiGsPp04+IaIpBPyO3quMSIyKiVNVxiZKOlLRG0h2Szunw+lskrZZ0i6RrJO3a7Z4J+hERFRsdHe16dSNpJnARcBSwF3CSpL3ast0ILLS9L3Al8P5u903Qj4iolMGj3a/uDgTusH2n7ceBzwHHblKSfa3tX5RPrwd27nbTBP2IiIp5Ev8BO0gaabkWtd1mJ+Duluf3lGnjOQP4l251y0BuRESFpjCQu9b2wirKlHQKsBD4vW55p2XQL78xFwHMnTu35tpERNNUNHvnXmCXluc7l2mbkPRy4K+B37P9WLebTsvuHduLbS+0vXDOnDl1VyciGqWYp9/tmoTlwHxJu0l6GnAisKQ1g6T9gYuBY2zfN5mbTsuWfkREnSYzO6cb2+slnQUsBWYCl9peJek8YMT2EuB8YFvgi+W5Az+2fcxE903Qj4ioUJWLs2xfBVzVlnZuy+OXT/WeQ929I+kqSc+tux4RERt54zm5E101GeqWvu1X1F2HiIh2JnvvREQ0xiDvvZOgHxFRKVcykNsrCfoRERXKcYkREQ2T7p2IiAZJ0I+IaIx6p2R2k6AfEVGxHIweEdEQNoyObqi7GuNK0I+IqNTkj0OsQ4J+RETFEvQjIhokQT8iokGyOCsioilq3kWzmwT9iIgKGRhNSz8iojnSvRMR0RiZshkR0SgJ+hERDVHlGbm90JMzciV9U9IaSTeV15Utry2SdHt5/UDSwS2vHS3pRkk3S1ot6XW9qF9ERO8Yj27oetWlspa+pKcBT7X9aJl0su2RtjxHA68DDra9VtIBwFckHQjcDywGDrR9j6SnA/PK9z3T9oNV1TUiopcGecO1LW7pS3q+pAuANcAeXbK/DTjb9loA2zcAnwTOBGZRfAndX772mO015ftOkHSrpL+QNGdL6xwR0Uu2u1512aygL+kZkk6T9G/AJcBqYF/bN7Zku6Kle+f8Mm1vYEXb7UaAvW0/ACwBfiTpHyWdLGkGgO2PA0cB2wDXSbpS0pFjr3eo3yJJI5JGfvazn23OR4yI2GyDHPQ3t3vnp8AtwGts3z5Onid173Rj+zWSXgC8HHgrcDhwavna3cA7Jb2L4gvgUoovjGM63GcxRVcRCxcuHNzfsyJi2imC+uDO09/c7p3jgXuBL0s6V9Kuk3zfamBBW9oCYNXYE9srbX+QIuAf15qx7Pv/KHAh8AXg7ZtX/YiI3hnklv5mBX3by2yfALwEeBj4Z0lXS5rX5a3vB94n6TcBJO1H0ZL/qKRtJR3aknc/4EdlviMk3QK8C7gW2Mv2n9leRUTEgBkdHe161WWLZu/Yvh/4MPDhshXeOg/pCkm/LB+vtf1y20sk7QR8V5KBdcAptn8qaRbwl5IuBn4JPErZtUMxuPsHtn+0JfWNiOiLAZ6nX9mUTds/aHl86AT5PgZ8rEP6OuAV47ynffA3ImJAGTO4ffrb7WIpAAABt0lEQVRZkRsRUaFBX5GboB8RUbEE/YiIBknQj4hoDDNa49463SToR0RUKH36ERFNM8BBvydbK0dENJcn9d9klHuMrZF0h6RzOrz+dEmfL1///iQWyE7/lv6KFSvWSprqoq4dgLW9qE/KGtpyUtZwlbW55Ux2S5kJVbH3jqSZwEUUW9LcAyyXtMT26pZsZwAP2v4tSScC7wNOmOi+0z7o257yVsySRmwv7EV9UtZwlpOyhqusfn6mTiraZuFA4A7bdwJI+hxwLMUeZmOOBf62fHwl8H8lyRMMKkz7oB8R0WdLKX7T6GYrSa07ES8udwgesxNwd8vze4AXtd3j13lsr5f0MPCbTPBbToJ+RESFbB9Zdx0mkoHczhZ3z5KyBqSs6fiZUtbwlNNL9wK7tDzfuUzrmEfSU4DtKU8fHI8GeT5pRERTlUH834GXUQT35cCrW7eUl3Qm8ALbry8Hcv/I9p9MdN9070REDKCyj/4sijGCmcCltldJOg8Ysb0E+ATwaUl3AA8AJ3a7b1r6ERENkj79iIgGSdCPiGiQBP2IiAZJ0I+IaJAE/YiIBknQj4hokAT9iIgG+f9C167KnziGOwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate_and_show_attention(\"c est un jeune directeur plein de talent .\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py3.6-env",
   "language": "python",
   "name": "py3.6-env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
